Sep 19 08:20:30.276357 ip-10-0-136-99 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:33.764424 ip-10-0-136-99 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:33.764424 ip-10-0-136-99 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:33.764424 ip-10-0-136-99 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:33.764424 ip-10-0-136-99 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:33.764424 ip-10-0-136-99 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:33.765210 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.764506    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:33.769128 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769112    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.769128 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769127    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769131    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769134    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769137    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769141    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769144    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769147    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769150    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769153    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769156    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769159    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769164    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769167    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769170    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769173    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769181    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769184    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769187    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769189    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.769193 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769192    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769195    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769198    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769201    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769204    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769207    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769210    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769212    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769215    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769219    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769223    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769226    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769229    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769231    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769234    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769237    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769240    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769243    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769245    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769248    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.769651 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769251    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769254    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769257    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769260    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769263    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769267    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769269    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769272    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769274    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769277    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769279    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769282    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769285    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769287    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769290    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769293    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769296    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769299    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769301    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769304    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.770182 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769306    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769309    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769312    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769314    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769316    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769319    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769321    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769324    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769327    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769329    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769332    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769335    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769338    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769340    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769343    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769346    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769348    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769351    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769353    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769356    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.770670 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769360    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769363    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769365    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769368    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769371    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769373    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769377    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769380    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769383    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769385    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769389    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769393    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769397    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769400    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769402    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769405    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769407    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769410    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.769412    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.771206 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770115    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770121    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770124    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770127    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770129    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770133    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770136    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770139    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770142    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770145    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770148    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770151    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770154    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770156    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770159    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770164    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770168    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770172    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.771683 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770175    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770178    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770182    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770185    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770188    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770191    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770195    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770197    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770200    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770203    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770205    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770208    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770211    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770213    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770216    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770219    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770222    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770225    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770227    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770230    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.772201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770238    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770241    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770243    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770246    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770249    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770252    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770255    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770258    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770260    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770265    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770268    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770271    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770274    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770277    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770280    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770282    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770285    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770288    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770290    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770293    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.772691 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770296    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770298    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770301    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770304    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770306    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770309    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770312    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770315    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770317    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770320    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770322    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770326    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770329    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770336    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770339    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770341    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770344    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770347    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770349    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770352    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.773201 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770354    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770357    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770360    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770363    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770365    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770368    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770370    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770373    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770376    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770378    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770381    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770383    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770386    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770388    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770391    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770394    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770396    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770399    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770402    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770405    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.773682 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770408    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770763    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770769    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770771    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770774    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770778    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770781    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770783    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770786    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770789    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770791    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770794    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770797    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770799    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770802    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770805    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770808    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770811    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770813    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.774185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770816    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770819    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770821    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770824    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770827    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770829    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770832    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770834    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770837    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770840    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770842    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770845    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770848    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770851    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770853    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770856    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770858    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770861    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770864    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770867    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.774642 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770870    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770872    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770875    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770877    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770880    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770882    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770885    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770887    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770890    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770893    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770895    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770898    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770900    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770905    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770908    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770912    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770915    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770917    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770920    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770922    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.775147 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770926    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770928    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770931    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770933    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770936    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770939    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770941    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770944    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770946    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770949    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770951    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770954    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770957    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770959    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770962    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770964    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770967    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770970    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770975    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770978    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.775640 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770995    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.770997    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771000    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771003    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771005    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771008    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771012    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771016    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771018    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771022    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771024    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771028    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771031    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771034    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771037    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771039    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771042    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771046    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771049    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.776176 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771052    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771055    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771129    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771160    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771166    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771170    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771175    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771178    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771183    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771187    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771190    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771195    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771198    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771202    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771205    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771208    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771211    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771214    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771217    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771220    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771224    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771227    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771230    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771233    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:33.776657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771236    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771240    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771243    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771246    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771249    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771254    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771257    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771260    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771263    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771266    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771270    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771273    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771276    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771279    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771282    2584 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771285    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771290    2584 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771293    2584 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771296    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771300    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771303    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771306    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771310    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771312    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771315    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:33.777257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771318    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771321    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771324    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771327    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771329    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771332    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771335    2584 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771339    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771342    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771345    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771348    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771351    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771355    2584 flags.go:64] FLAG: --help="false"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771358    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-136-99.ec2.internal"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771361    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771364    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771367    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771371    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771374    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771377    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771380    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771383    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771386    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:33.777888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771389    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771392    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771395    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771399    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771402    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771405    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771408    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771411    2584 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771413    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771416    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771419    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771425    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771427    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771430    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771433    2584 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771436    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771439    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771442    2584 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771445    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771450    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771453    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771458    2584 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771461    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771464    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771467    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:33.778483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771470    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771473    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771476    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771479    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771485    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771489    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771492    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771494    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771498    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771503    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771506    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771511    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771514    2584 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771517    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771519    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008a1d45bd190ad34"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771523    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771526    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771529    2584 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771531    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771535    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771538    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771541    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771543    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771547    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771550    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:33.779166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771553    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771556    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771559    2584 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771563    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771566    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771569    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771572    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771575    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771578    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771581    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771584    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771586    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771589    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771592    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771595    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771598    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771601    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771604    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771611    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771614    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771617    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771621    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771624    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771626    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771629    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:33.779785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771632    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771635    2584 flags.go:64] FLAG: --v="2"
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771639    2584 flags.go:64] FLAG: --version="false"
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771642    2584 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771646    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.771649    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771763    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771767    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771770    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771773    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771777    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771780    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771783    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771785    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771788    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771790    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771793    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771795    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771798    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771801    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771803    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.780408 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771806    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771809    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771811    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771814    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771818    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771821    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771824    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771827    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771829    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771833    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771837    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771839    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771842    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771845    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771847    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771851    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771855    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771858    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.780959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771860    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771863    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771865    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771882    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771886    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771891    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771894    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771896    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771899    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771902    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771905    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771907    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771910    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771912    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771915    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771918    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771921    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771924    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771927    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771930    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.781417 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771933    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771935    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771938    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771940    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771943    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771946    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771948    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771951    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771954    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771957    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771960    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771962    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771965    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771968    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771970    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771974    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771977    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771992    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771995    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.771997    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.781896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772000    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772003    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772006    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772009    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772012    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772015    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772017    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772020    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772023    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772025    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772028    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772031    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772034    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772036    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772039    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772041    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772044    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772047    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772049    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772052    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.782392 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772054    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772057    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772059    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772062    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772066    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.772069    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.772867    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.780447    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.780461    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.780478    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.780484    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780535    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780541    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780544    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.782871 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780547    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780550    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780552    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780555    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780559    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780562    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780565    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780568    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780571    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780574    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780577    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780580    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780585    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780588    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780591    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780594    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780597    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780601    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780604    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.783235 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780607    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780611    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780614    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780617    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780620    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780624    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780627    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780631    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780634    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780637    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780639    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780642    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780644    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780648    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780650    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780653    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780656    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780658    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780661    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.783762 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780664    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780667    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780669    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780672    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780674    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780677    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780680    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780682    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780685    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780688    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780691    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780693    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780696    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780699    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780701    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780704    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780706    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780709    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780712    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780715    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.784346 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780718    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780721    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780724    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780727    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780729    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780732    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780734    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780737    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780740    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780743    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780745    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780748    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780750    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780753    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780756    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780759    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780761    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780764    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780766    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780769    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.784888 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780772    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780774    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780777    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780780    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780782    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780785    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780788    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780790    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780793    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780795    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780798    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780801    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780804    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780806    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780809    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780812    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780814    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.785416 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780817    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.780822    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780932    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780937    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780940    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780943    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780946    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780950    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780952    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780955    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780958    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780961    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780964    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780967    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780970    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:33.785834 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780973    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780976    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780992    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780995    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.780998    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781001    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781003    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781006    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781009    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781011    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781014    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781017    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781020    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781023    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781026    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781029    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781032    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781035    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781038    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781040    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:33.786238 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781043    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781046    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781048    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781051    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781053    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781056    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781060    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781062    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781065    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781068    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781070    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781073    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781076    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781078    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781081    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781083    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781086    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781089    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781091    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:33.786735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781095    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781099    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781102    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781105    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781108    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781111    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781114    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781117    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781119    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781123    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781127    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781129    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781132    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781134    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781137    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781139    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781142    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781144    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781147    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781150    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:33.787350 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781152    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781155    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781158    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781161    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781164    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781167    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781170    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781173    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781175    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781178    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781180    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781183    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781185    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781188    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781190    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781193    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781196    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781199    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781201    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:33.787837 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781204    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781207    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781209    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781212    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781215    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781217    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781220    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:33.781222    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.781227    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.781328    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.783753    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.784831    2584 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.784927    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:33.788550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.784961    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:33.789831 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.789816    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:33.790913 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.790896    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:33.804270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.804250    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:33.805826 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.805806    2584 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:33.806658 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.806634    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:33.809675 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.809659    2584 fs.go:135] Filesystem UUIDs: map[64bd7ffc-81fd-4528-858a-eb4f8b536494:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a7a8275a-fa6e-4f7e-ba68-ef677f6bb30b:/dev/nvme0n1p3]
Sep 19 08:20:33.809731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.809675    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:33.813846 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.813743    2584 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:33.813250138 +0000 UTC m=+1.897750932 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098933 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d175230f11224494c8231e4cfb88f SystemUUID:ec2d1752-30f1-1224-494c-8231e4cfb88f BootID:65431c28-300f-42e3-ad0a-5964899ee1dc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:6f:4f:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:6f:4f:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:44:e5:0f:58:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:33.813846 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.813842    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:33.813957 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.813943    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:33.815404 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.815379    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:33.815525 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.815407    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:33.816448 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.816436    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:33.816486 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.816451    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:33.816486 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.816459    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:33.816486 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.816474    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:33.817631 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.817621    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:33.821719 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821707    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:33.821842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821833    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:33.821871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821861    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:33.821898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821876    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:33.821898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821890    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:33.821898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821898    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:33.822018 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.821908    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:33.822253 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.822240    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:33.822324 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.822258    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:33.823298 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.823276    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:33.824085 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.824065    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:33.827020 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.827000    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:33.827086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.827038    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:33.829139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829113    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829145    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829156    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829161    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829170    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829193    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829206    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829212    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829219    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:33.829220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829225    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:33.829484 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829233    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:33.829484 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829239    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:33.829484 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829262    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:33.829484 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.829267    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:33.833216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.833200    2584 server.go:1268] "Started kubelet"
Sep 19 08:20:33.833333 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.833267    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:33.833392 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.833339    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:33.833448 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.833411    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:33.834043 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.833707    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:33.834057 ip-10-0-136-99 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:33.834593 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.834160    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:33.834593 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.834426    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:33.834593 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.834484    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:33.834593 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.834491    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:33.839088 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.839064    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:33.840332 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.840311    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:33.840332 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.840324    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:33.840496 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.840484    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:33.841176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841158    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:33.841176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841159    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841198    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841167    2584 factory.go:153] Registering CRI-O factory
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841224    2584 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841277    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841286    2584 factory.go:55] Registering systemd factory
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841293    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:33.841330 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841313    2584 factory.go:103] Registering Raw factory
Sep 19 08:20:33.841598 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841339    2584 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:33.841598 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841361    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:33.841598 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841372    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:33.841769 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.841757    2584 manager.go:349] Starting recovery of all containers
Sep 19 08:20:33.842883 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.842852    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 08:20:33.843005 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.842943    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 08:20:33.852685 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.852671    2584 manager.go:354] Recovery completed
Sep 19 08:20:33.875695 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.875673    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:33.878235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878214    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:33.878327 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878262    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:33.878327 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878279    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:33.878814 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878800    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:33.878814 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878811    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:33.878934 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878821    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:33.878934 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.878824    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:33.879507 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.879492    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:33.879586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.879514    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:33.879586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.879524    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:33.880961 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.880948    2584 policy_none.go:49] "None policy: Start"
Sep 19 08:20:33.880961 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.880963    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:33.881069 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.880973    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:33.882927 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.882916    2584 policy_none.go:47] "Start"
Sep 19 08:20:33.886180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.886095    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nn6h"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.887443    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-99.ec2.internal.18d6ab1adf7a25ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-99.ec2.internal,UID:ip-10-0-136-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-99.ec2.internal,},FirstTimestamp:2026-09-19 08:20:33.833174443 +0000 UTC m=+1.917675237,LastTimestamp:2026-09-19 08:20:33.833174443 +0000 UTC m=+1.917675237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-99.ec2.internal,}"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.891052    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-99.ec2.internal.18d6ab1ae229b48e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-99.ec2.internal,UID:ip-10-0-136-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-99.ec2.internal,},FirstTimestamp:2026-09-19 08:20:33.878234254 +0000 UTC m=+1.962735053,LastTimestamp:2026-09-19 08:20:33.878234254 +0000 UTC m=+1.962735053,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-99.ec2.internal,}"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.894551    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nn6h"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.899051    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.899435    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-99.ec2.internal.18d6ab1ae22a4138  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-99.ec2.internal,UID:ip-10-0-136-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-99.ec2.internal,},FirstTimestamp:2026-09-19 08:20:33.878270264 +0000 UTC m=+1.962771058,LastTimestamp:2026-09-19 08:20:33.878270264 +0000 UTC m=+1.962771058,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-99.ec2.internal,}"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.900364    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.900431    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.900456    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.900500    2584 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 19 08:20:33.914571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.903049    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:33.917514 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917502    2584 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:33.917572 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.917533    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:33.917572 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917542    2584 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:33.917782 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917770    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:33.917821 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917791    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:33.917893 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917874    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:33.917973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917957    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:33.917973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:33.917970    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:33.918401 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.918378    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:33.918466 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:33.918435    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.001538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.001515    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"]
Sep 19 08:20:34.001599 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.001573    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.002512 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.002499    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.002589 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.002522    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.002589 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.002532    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.004716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.004703    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.004837 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.004814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.004884 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.004851    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.006106 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006089    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.006174 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006117    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.006174 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006127    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.006239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006089    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.006239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006201    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.006239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.006213    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.008325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.008310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.008382 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.008335    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.009106 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.009092    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.009171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.009115    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.009171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.009128    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.018285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.018237    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.018962 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.018949    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.019041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.018971    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.019041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.018999    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.019041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.019030    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.027541 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.027527    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-99.ec2.internal\" not found" node="ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.028640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.028623    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.028694 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.028651    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-99.ec2.internal\": node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.031498 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.031483    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-99.ec2.internal\" not found" node="ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.043380 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.043360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3207445c8278099d149a5cf6a8c10f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"cc3207445c8278099d149a5cf6a8c10f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.043451 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.043383    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.043451 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.043398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.074650 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.074631    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.143907 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.143879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.143968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.143904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.143968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.143922    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.143968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.143939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3207445c8278099d149a5cf6a8c10f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"cc3207445c8278099d149a5cf6a8c10f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.144092 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.143971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3207445c8278099d149a5cf6a8c10f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-99.ec2.internal\" (UID: \"cc3207445c8278099d149a5cf6a8c10f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.144092 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.144017    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff625bfa14bbf75c8218eec2f16b1f24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal\" (UID: \"ff625bfa14bbf75c8218eec2f16b1f24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.175470 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.175451    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.276016 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.275962    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.330406 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.330387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.335842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.335391    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:34.376514 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.376493    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.477054 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.477027    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.577528 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.577467    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.678008 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.677974    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.702804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.702781    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:34.778452 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.778429    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.784723 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.784576    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:34.784890 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.784862    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:34.784943 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.784885    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:34.878547 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.878526    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:34.884380 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:34.884349    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3207445c8278099d149a5cf6a8c10f.slice/crio-7b2e73ca49ec4ae592dd0f7bd026c6c0f2e5517bf5b6a5e288d4300cd4c5b5a7 WatchSource:0}: Error finding container 7b2e73ca49ec4ae592dd0f7bd026c6c0f2e5517bf5b6a5e288d4300cd4c5b5a7: Status 404 returned error can't find the container with id 7b2e73ca49ec4ae592dd0f7bd026c6c0f2e5517bf5b6a5e288d4300cd4c5b5a7
Sep 19 08:20:34.884861 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:34.884837    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff625bfa14bbf75c8218eec2f16b1f24.slice/crio-0075da1a7c3fbe97e43190c8b846a039724fa613135b4686be18e3e14f3a608a WatchSource:0}: Error finding container 0075da1a7c3fbe97e43190c8b846a039724fa613135b4686be18e3e14f3a608a: Status 404 returned error can't find the container with id 0075da1a7c3fbe97e43190c8b846a039724fa613135b4686be18e3e14f3a608a
Sep 19 08:20:34.889960 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.889947    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:34.899432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.899402    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:33 +0000 UTC" deadline="2028-03-19 19:14:39.612997374 +0000 UTC"
Sep 19 08:20:34.899432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.899430    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13138h54m4.713569874s"
Sep 19 08:20:34.903758 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.903716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" event={"ID":"cc3207445c8278099d149a5cf6a8c10f","Type":"ContainerStarted","Data":"7b2e73ca49ec4ae592dd0f7bd026c6c0f2e5517bf5b6a5e288d4300cd4c5b5a7"}
Sep 19 08:20:34.904779 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:34.904761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"ff625bfa14bbf75c8218eec2f16b1f24","Type":"ContainerStarted","Data":"0075da1a7c3fbe97e43190c8b846a039724fa613135b4686be18e3e14f3a608a"}
Sep 19 08:20:34.979420 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:34.979389    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:35.036542 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.036524    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:35.037401 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.037386    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:35.037464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.037411    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:35.037464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.037421    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:35.080034 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.080003    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:35.180912 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.180862    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:35.281270 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.281237    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-99.ec2.internal\" not found"
Sep 19 08:20:35.302857 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.302795    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:35.374814 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.374662    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:35.440794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.440724    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:35.453456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.453338    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:35.454531 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.454344    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal"
Sep 19 08:20:35.462759 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.462740    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:35.823025 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.822937    2584 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:35.830357 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.830333    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:35.830737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.830714    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2lpmf","openshift-network-operator/iptables-alerter-qgjpr","openshift-ovn-kubernetes/ovnkube-node-sbhch","kube-system/global-pull-secret-syncer-j6tqd","kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474","openshift-cluster-node-tuning-operator/tuned-9h2kp","openshift-dns/node-resolver-p2tqn","openshift-image-registry/node-ca-lmzgn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal","openshift-multus/multus-5w7k7","kube-system/konnectivity-agent-vbq2l","openshift-multus/multus-additional-cni-plugins-r62dt","openshift-multus/network-metrics-daemon-s4btv"]
Sep 19 08:20:35.833279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.833251    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.835521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.835498    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.835675 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.835655    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whg67\""
Sep 19 08:20:35.835772 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.835663    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.836109 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.836090    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.837725 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.837705    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.838465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.838043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.838465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.838110    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:35.838465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.838232    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:35.838465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.838252    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fhlwh\""
Sep 19 08:20:35.838620 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.838478    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.840089 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.840064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.840175 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.840136    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:35.840829 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.840810    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:35.840910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.840831    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.840964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.840909    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:35.841369 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.841071    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:35.841369 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.841205    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.841369 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.841209    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f4fzs\""
Sep 19 08:20:35.841369 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.841311    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:35.842285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.842269    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:35.842576 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.842557    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.842657 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.842627    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:35.844900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.844880    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.847058 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.847036    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skrk7\""
Sep 19 08:20:35.847165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.847084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:35.847267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.847251    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.847336 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.847268    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.847697 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.847676    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:35.849829 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.849792    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.850078 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.850045    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:35.850078 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.850051    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b5575\""
Sep 19 08:20:35.850303 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.850286    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.850303 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.850296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.852332 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.852315    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:35.852436 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.852423    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:35.852688 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.852674    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:35.853086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853098    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-ovn\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-cnibin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-conf-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-kubelet\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-tuned\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853235    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-multus\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-dbus\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853317    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853314    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ghbb8\""
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.853396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-kubernetes\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-kubelet\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-etc-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-env-overrides\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-var-lib-kubelet\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-k8s-cni-cncf-io\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-multus-daemon-config\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-script-lib\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.853676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysconfig\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853582    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-tmp\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853920    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-config\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.853948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcr9r\" (UniqueName: \"kubernetes.io/projected/d0b18af0-056d-42a2-aee2-e0fb3d43a166-kube-api-access-hcr9r\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-socket-dir-parent\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-bin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-run\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-sys\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-system-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-multus-certs\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854167    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-systemd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-kubelet-config\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-modprobe-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zrv\" (UniqueName: \"kubernetes.io/projected/bd450284-312e-40a7-9188-8a0692c0b872-kube-api-access-54zrv\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-slash\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwp5\" (UniqueName: \"kubernetes.io/projected/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-kube-api-access-cgwp5\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-cni-binary-copy\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-netns\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854309    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-etc-kubernetes\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854331    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-bin\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854363    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dcf\" (UniqueName: \"kubernetes.io/projected/5c2aa2ce-a894-4087-b089-83fa0a5a824c-kube-api-access-r2dcf\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854381    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-conf\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.854444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-systemd\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854466    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-log-socket\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-netd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.854627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovn-node-metrics-cert\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-lib-modules\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855143    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-host\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.855233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-hostroot\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.855490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-systemd-units\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855335    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-netns\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-var-lib-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-node-log\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855666 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.855666 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.855555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.857594 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.856419    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:35.858144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.858009    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.858144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.858065    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.858376 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.858354    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:35.859462 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.859443    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nfld6\""
Sep 19 08:20:35.859564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.859547    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:35.859653 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.859625    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:35.859724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.859669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:35.859724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.859676    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5c2vz\""
Sep 19 08:20:35.859841 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.859745    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:35.862199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.862181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:35.864235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.864217    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:35.864482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.864406    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:35.864482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.864451    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x468b\""
Sep 19 08:20:35.876017 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.875993    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hm9mh"
Sep 19 08:20:35.883196 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.883171    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hm9mh"
Sep 19 08:20:35.887714 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.887697    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:35.942319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.942290    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:35.956340 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.956467 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.956368    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:35.956467 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028b379e-a3d7-4094-908c-e51d0ce4d1ff-host\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:35.956467 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.956460    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.456429352 +0000 UTC m=+4.540930150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956484    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-ovn\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be6872e-0257-4d2a-8943-eec60b49bd88-hosts-file\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be6872e-0257-4d2a-8943-eec60b49bd88-tmp-dir\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-cnibin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.956640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-conf-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-ovn\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-kubelet\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mr4\" (UniqueName: \"kubernetes.io/projected/028b379e-a3d7-4094-908c-e51d0ce4d1ff-kube-api-access-86mr4\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-kubelet\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-conf-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-cnibin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-host-slash\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-tuned\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956839    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-multus\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-dbus\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956893    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-kubernetes\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-kubelet\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-etc-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.956973    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-multus\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957009    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-env-overrides\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-etc-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957095 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.956974    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-kubernetes\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-dbus\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:35.957125    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.457099721 +0000 UTC m=+4.541600527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-kubelet\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-var-lib-kubelet\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-k8s-cni-cncf-io\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-multus-daemon-config\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957236    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-var-lib-kubelet\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-k8s-cni-cncf-io\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-script-lib\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957278    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysconfig\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-tmp\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.957910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-config\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957399    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcr9r\" (UniqueName: \"kubernetes.io/projected/d0b18af0-056d-42a2-aee2-e0fb3d43a166-kube-api-access-hcr9r\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957424    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-socket-dir-parent\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-socket-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957452    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7q9\" (UniqueName: \"kubernetes.io/projected/1be7dfc1-db2c-4a06-98e4-478250e634a9-kube-api-access-xn7q9\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htp5x\" (UniqueName: \"kubernetes.io/projected/0be6872e-0257-4d2a-8943-eec60b49bd88-kube-api-access-htp5x\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028b379e-a3d7-4094-908c-e51d0ce4d1ff-serviceca\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-env-overrides\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-socket-dir-parent\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-bin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957657    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957680    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-konnectivity-ca\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957712    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-multus-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.958710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-run\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-registration-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957765    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-var-lib-cni-bin\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-multus-daemon-config\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klv4\" (UniqueName: \"kubernetes.io/projected/9920c568-66e7-43c9-839d-8b8846d5d8c2-kube-api-access-7klv4\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-script-lib\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt572\" (UniqueName: \"kubernetes.io/projected/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-kube-api-access-bt572\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-run\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957842    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-sys\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-system-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-multus-certs\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysconfig\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957937    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-systemd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-sys\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.957974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-kubelet-config\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958005    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-run-systemd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.959504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-sys-fs\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958028    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-system-cni-dir\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-multus-certs\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-modprobe-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4775e37-d3b3-414d-a623-177de1acbfc0-kubelet-config\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zrv\" (UniqueName: \"kubernetes.io/projected/bd450284-312e-40a7-9188-8a0692c0b872-kube-api-access-54zrv\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-slash\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958135    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-device-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-modprobe-d\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwp5\" (UniqueName: \"kubernetes.io/projected/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-kube-api-access-cgwp5\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-cni-binary-copy\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-slash\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958193    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovnkube-config\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-netns\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958282    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-host-run-netns\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-etc-kubernetes\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.960372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-bin\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dcf\" (UniqueName: \"kubernetes.io/projected/5c2aa2ce-a894-4087-b089-83fa0a5a824c-kube-api-access-r2dcf\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-etc-kubernetes\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-agent-certs\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-conf\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-bin\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-systemd\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-log-socket\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-netd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovn-node-metrics-cert\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958731    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd450284-312e-40a7-9188-8a0692c0b872-cni-binary-copy\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958739    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-iptables-alerter-script\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-lib-modules\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-host\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-hostroot\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.961264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-systemd-units\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-sysctl-conf\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-cnibin\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-log-socket\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958874    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-systemd\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958923    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-netns\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958926    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-host\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.958972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-cni-netd\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-var-lib-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-run-netns\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-node-log\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959062    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-var-lib-openvswitch\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-node-log\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-lib-modules\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959111    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-systemd-units\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd450284-312e-40a7-9188-8a0692c0b872-hostroot\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.961830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.959154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2aa2ce-a894-4087-b089-83fa0a5a824c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.962411 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.961522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-etc-tuned\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.962411 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.961607    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-tmp\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.962411 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.961728    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2aa2ce-a894-4087-b089-83fa0a5a824c-ovn-node-metrics-cert\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:35.975390 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.974150    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcr9r\" (UniqueName: \"kubernetes.io/projected/d0b18af0-056d-42a2-aee2-e0fb3d43a166-kube-api-access-hcr9r\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:35.975590 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.975566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwp5\" (UniqueName: \"kubernetes.io/projected/3a2b6c98-8849-4a6f-87a6-648d889e4b4f-kube-api-access-cgwp5\") pod \"tuned-9h2kp\" (UID: \"3a2b6c98-8849-4a6f-87a6-648d889e4b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:35.976848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.976780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zrv\" (UniqueName: \"kubernetes.io/projected/bd450284-312e-40a7-9188-8a0692c0b872-kube-api-access-54zrv\") pod \"multus-5w7k7\" (UID: \"bd450284-312e-40a7-9188-8a0692c0b872\") " pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:35.977375 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:35.977353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dcf\" (UniqueName: \"kubernetes.io/projected/5c2aa2ce-a894-4087-b089-83fa0a5a824c-kube-api-access-r2dcf\") pod \"ovnkube-node-sbhch\" (UID: \"5c2aa2ce-a894-4087-b089-83fa0a5a824c\") " pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:36.060347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:36.060541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.060541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.060541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-socket-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.060541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060520    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-socket-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7q9\" (UniqueName: \"kubernetes.io/projected/1be7dfc1-db2c-4a06-98e4-478250e634a9-kube-api-access-xn7q9\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htp5x\" (UniqueName: \"kubernetes.io/projected/0be6872e-0257-4d2a-8943-eec60b49bd88-kube-api-access-htp5x\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028b379e-a3d7-4094-908c-e51d0ce4d1ff-serviceca\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060708    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.060734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-konnectivity-ca\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060778    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-registration-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klv4\" (UniqueName: \"kubernetes.io/projected/9920c568-66e7-43c9-839d-8b8846d5d8c2-kube-api-access-7klv4\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060828    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt572\" (UniqueName: \"kubernetes.io/projected/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-kube-api-access-bt572\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060858    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-sys-fs\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060860    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-sys-fs\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.060965    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-registration-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061443 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061419    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/028b379e-a3d7-4094-908c-e51d0ce4d1ff-serviceca\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-device-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061487    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-agent-certs\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-iptables-alerter-script\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.061567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be7dfc1-db2c-4a06-98e4-478250e634a9-device-dir\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061572    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-cnibin\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028b379e-a3d7-4094-908c-e51d0ce4d1ff-host\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-konnectivity-ca\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be6872e-0257-4d2a-8943-eec60b49bd88-hosts-file\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be6872e-0257-4d2a-8943-eec60b49bd88-tmp-dir\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mr4\" (UniqueName: \"kubernetes.io/projected/028b379e-a3d7-4094-908c-e51d0ce4d1ff-kube-api-access-86mr4\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-host-slash\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061689    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-system-cni-dir\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061826    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-host-slash\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061837    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.061867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/028b379e-a3d7-4094-908c-e51d0ce4d1ff-host\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.062398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.061995    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9920c568-66e7-43c9-839d-8b8846d5d8c2-cnibin\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.062398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.062055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be6872e-0257-4d2a-8943-eec60b49bd88-hosts-file\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.062398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.062298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-iptables-alerter-script\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.062398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.062312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be6872e-0257-4d2a-8943-eec60b49bd88-tmp-dir\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.062398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.062322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9920c568-66e7-43c9-839d-8b8846d5d8c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.063696 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.063669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171bdcd-0b75-4c34-8d84-9c28cfd810a0-agent-certs\") pod \"konnectivity-agent-vbq2l\" (UID: \"6171bdcd-0b75-4c34-8d84-9c28cfd810a0\") " pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:36.066156 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.066141    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:36.066237 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.066161    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:36.066237 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.066170    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.066237 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.066216    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.566204931 +0000 UTC m=+4.650705712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.072428 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.072395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt572\" (UniqueName: \"kubernetes.io/projected/b50d53d5-1085-4aea-8b4c-bcdcf003d4b5-kube-api-access-bt572\") pod \"iptables-alerter-qgjpr\" (UID: \"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5\") " pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.073050 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.073005    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7q9\" (UniqueName: \"kubernetes.io/projected/1be7dfc1-db2c-4a06-98e4-478250e634a9-kube-api-access-xn7q9\") pod \"aws-ebs-csi-driver-node-z7474\" (UID: \"1be7dfc1-db2c-4a06-98e4-478250e634a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.073120 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.073072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mr4\" (UniqueName: \"kubernetes.io/projected/028b379e-a3d7-4094-908c-e51d0ce4d1ff-kube-api-access-86mr4\") pod \"node-ca-lmzgn\" (UID: \"028b379e-a3d7-4094-908c-e51d0ce4d1ff\") " pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.073171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.073156    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htp5x\" (UniqueName: \"kubernetes.io/projected/0be6872e-0257-4d2a-8943-eec60b49bd88-kube-api-access-htp5x\") pod \"node-resolver-p2tqn\" (UID: \"0be6872e-0257-4d2a-8943-eec60b49bd88\") " pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.073239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.073225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klv4\" (UniqueName: \"kubernetes.io/projected/9920c568-66e7-43c9-839d-8b8846d5d8c2-kube-api-access-7klv4\") pod \"multus-additional-cni-plugins-r62dt\" (UID: \"9920c568-66e7-43c9-839d-8b8846d5d8c2\") " pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.145167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.145139    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9h2kp"
Sep 19 08:20:36.152009 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.151989    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w7k7"
Sep 19 08:20:36.161699 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.161682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:20:36.168262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.168244    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474"
Sep 19 08:20:36.173838 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.173814    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lmzgn"
Sep 19 08:20:36.180529 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.180514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r62dt"
Sep 19 08:20:36.187011 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.186997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgjpr"
Sep 19 08:20:36.193469 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.193457    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2tqn"
Sep 19 08:20:36.198965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.198951    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:36.464675 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.464601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:36.464675 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.464653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:36.464891 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.464753    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:36.464891 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.464768    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:36.464891 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.464817    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.464802587 +0000 UTC m=+5.549303368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:36.464891 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.464834    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.464827818 +0000 UTC m=+5.549328600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:36.666590 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.666559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:36.666744 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.666724    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:36.666798 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.666749    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:36.666798 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.666763    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.666903 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:36.666814    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.666801592 +0000 UTC m=+5.751302373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.725803 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.725693    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9920c568_66e7_43c9_839d_8b8846d5d8c2.slice/crio-df03b094b1ee9d0e2443a0dfc1986c2d0f02da5888d345e5c7a82de924254921 WatchSource:0}: Error finding container df03b094b1ee9d0e2443a0dfc1986c2d0f02da5888d345e5c7a82de924254921: Status 404 returned error can't find the container with id df03b094b1ee9d0e2443a0dfc1986c2d0f02da5888d345e5c7a82de924254921
Sep 19 08:20:36.727306 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.727285    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd450284_312e_40a7_9188_8a0692c0b872.slice/crio-d64141cf1042f627d218b4c86d2d0d857ec8a65ec576c9d5535c807ec3ff0638 WatchSource:0}: Error finding container d64141cf1042f627d218b4c86d2d0d857ec8a65ec576c9d5535c807ec3ff0638: Status 404 returned error can't find the container with id d64141cf1042f627d218b4c86d2d0d857ec8a65ec576c9d5535c807ec3ff0638
Sep 19 08:20:36.728530 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.728505    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2aa2ce_a894_4087_b089_83fa0a5a824c.slice/crio-eed66772ae8f0fa262b03bc642729a22174b719180f558f1a361f3702fdacd54 WatchSource:0}: Error finding container eed66772ae8f0fa262b03bc642729a22174b719180f558f1a361f3702fdacd54: Status 404 returned error can't find the container with id eed66772ae8f0fa262b03bc642729a22174b719180f558f1a361f3702fdacd54
Sep 19 08:20:36.730258 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.730223    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6171bdcd_0b75_4c34_8d84_9c28cfd810a0.slice/crio-3723c357371eef25bdafbd2eabbf26345cdbeaf1a13d79cdd515783f1192e028 WatchSource:0}: Error finding container 3723c357371eef25bdafbd2eabbf26345cdbeaf1a13d79cdd515783f1192e028: Status 404 returned error can't find the container with id 3723c357371eef25bdafbd2eabbf26345cdbeaf1a13d79cdd515783f1192e028
Sep 19 08:20:36.732436 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.732273    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50d53d5_1085_4aea_8b4c_bcdcf003d4b5.slice/crio-781b3650ce592529a5179017eae55ac8b5fdd09be00666b2254911e4c65f3b9f WatchSource:0}: Error finding container 781b3650ce592529a5179017eae55ac8b5fdd09be00666b2254911e4c65f3b9f: Status 404 returned error can't find the container with id 781b3650ce592529a5179017eae55ac8b5fdd09be00666b2254911e4c65f3b9f
Sep 19 08:20:36.733237 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.733218    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028b379e_a3d7_4094_908c_e51d0ce4d1ff.slice/crio-0463eb1ddea990cd9d4837ef43b01042ee003a0d9c0e295441402eec434e61e7 WatchSource:0}: Error finding container 0463eb1ddea990cd9d4837ef43b01042ee003a0d9c0e295441402eec434e61e7: Status 404 returned error can't find the container with id 0463eb1ddea990cd9d4837ef43b01042ee003a0d9c0e295441402eec434e61e7
Sep 19 08:20:36.734746 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.734700    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be6872e_0257_4d2a_8943_eec60b49bd88.slice/crio-bc7a0c78db38e3800c763d4a63c7f2e05de6656a71d4adaf0c4f29c9d9f01a1d WatchSource:0}: Error finding container bc7a0c78db38e3800c763d4a63c7f2e05de6656a71d4adaf0c4f29c9d9f01a1d: Status 404 returned error can't find the container with id bc7a0c78db38e3800c763d4a63c7f2e05de6656a71d4adaf0c4f29c9d9f01a1d
Sep 19 08:20:36.735963 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.735945    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be7dfc1_db2c_4a06_98e4_478250e634a9.slice/crio-560bff94bde1c074b845c8b0dbd158b62a3361729129405829db5fffc3a4bd75 WatchSource:0}: Error finding container 560bff94bde1c074b845c8b0dbd158b62a3361729129405829db5fffc3a4bd75: Status 404 returned error can't find the container with id 560bff94bde1c074b845c8b0dbd158b62a3361729129405829db5fffc3a4bd75
Sep 19 08:20:36.738068 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:20:36.737860    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2b6c98_8849_4a6f_87a6_648d889e4b4f.slice/crio-e6b48581732fc4c705161228f932b11e5dbed8a72d70b1ee1b0cdb5563042522 WatchSource:0}: Error finding container e6b48581732fc4c705161228f932b11e5dbed8a72d70b1ee1b0cdb5563042522: Status 404 returned error can't find the container with id e6b48581732fc4c705161228f932b11e5dbed8a72d70b1ee1b0cdb5563042522
Sep 19 08:20:36.886244 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.886197    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:35 +0000 UTC" deadline="2028-05-20 17:07:21.169265533 +0000 UTC"
Sep 19 08:20:36.886244 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.886240    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14624h46m44.283027664s"
Sep 19 08:20:36.908871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.908841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbq2l" event={"ID":"6171bdcd-0b75-4c34-8d84-9c28cfd810a0","Type":"ContainerStarted","Data":"3723c357371eef25bdafbd2eabbf26345cdbeaf1a13d79cdd515783f1192e028"}
Sep 19 08:20:36.910554 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.910529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerStarted","Data":"df03b094b1ee9d0e2443a0dfc1986c2d0f02da5888d345e5c7a82de924254921"}
Sep 19 08:20:36.912203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.912175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmzgn" event={"ID":"028b379e-a3d7-4094-908c-e51d0ce4d1ff","Type":"ContainerStarted","Data":"0463eb1ddea990cd9d4837ef43b01042ee003a0d9c0e295441402eec434e61e7"}
Sep 19 08:20:36.913332 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.913310    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"eed66772ae8f0fa262b03bc642729a22174b719180f558f1a361f3702fdacd54"}
Sep 19 08:20:36.914415 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.914393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w7k7" event={"ID":"bd450284-312e-40a7-9188-8a0692c0b872","Type":"ContainerStarted","Data":"d64141cf1042f627d218b4c86d2d0d857ec8a65ec576c9d5535c807ec3ff0638"}
Sep 19 08:20:36.916172 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.916151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" event={"ID":"cc3207445c8278099d149a5cf6a8c10f","Type":"ContainerStarted","Data":"3bb3204074bd133d0fb845bddf0448e50dbbe9168ea9fc7f9c7c5a23f227e005"}
Sep 19 08:20:36.917402 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.917363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9h2kp" event={"ID":"3a2b6c98-8849-4a6f-87a6-648d889e4b4f","Type":"ContainerStarted","Data":"e6b48581732fc4c705161228f932b11e5dbed8a72d70b1ee1b0cdb5563042522"}
Sep 19 08:20:36.918548 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.918525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474" event={"ID":"1be7dfc1-db2c-4a06-98e4-478250e634a9","Type":"ContainerStarted","Data":"560bff94bde1c074b845c8b0dbd158b62a3361729129405829db5fffc3a4bd75"}
Sep 19 08:20:36.919582 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.919564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2tqn" event={"ID":"0be6872e-0257-4d2a-8943-eec60b49bd88","Type":"ContainerStarted","Data":"bc7a0c78db38e3800c763d4a63c7f2e05de6656a71d4adaf0c4f29c9d9f01a1d"}
Sep 19 08:20:36.920480 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.920455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgjpr" event={"ID":"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5","Type":"ContainerStarted","Data":"781b3650ce592529a5179017eae55ac8b5fdd09be00666b2254911e4c65f3b9f"}
Sep 19 08:20:36.945210 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:36.945166    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-99.ec2.internal" podStartSLOduration=1.94515495 podStartE2EDuration="1.94515495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:36.944876529 +0000 UTC m=+5.029377332" watchObservedRunningTime="2026-09-19 08:20:36.94515495 +0000 UTC m=+5.029655753"
Sep 19 08:20:37.473396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.473346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:37.473568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.473420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:37.473568 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.473544    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:37.473693 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.473606    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.473586663 +0000 UTC m=+7.558087462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:37.474044 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.474022    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:37.474151 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.474076    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.474062618 +0000 UTC m=+7.558563403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:37.676122 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.675504    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:37.676122 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.675668    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:37.676122 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.675689    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:37.676122 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.675702    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:37.676122 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.675757    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:20:39.67574004 +0000 UTC m=+7.760240825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:37.693648 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.693620    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:37.886670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.886559    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:35 +0000 UTC" deadline="2028-04-16 18:34:54.779613261 +0000 UTC"
Sep 19 08:20:37.886670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.886594    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13810h14m16.893023542s"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.903707    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.903821    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.904227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.904347    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.904424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:37.904564 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:37.904511    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:37.928247 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.928218    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff625bfa14bbf75c8218eec2f16b1f24" containerID="c3f1fe7a0aadc827e1cd78ccc568b2751871560d15afef3f829a5fe638c7cf13" exitCode=0
Sep 19 08:20:37.928369 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:37.928306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"ff625bfa14bbf75c8218eec2f16b1f24","Type":"ContainerDied","Data":"c3f1fe7a0aadc827e1cd78ccc568b2751871560d15afef3f829a5fe638c7cf13"}
Sep 19 08:20:38.941765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:38.941105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" event={"ID":"ff625bfa14bbf75c8218eec2f16b1f24","Type":"ContainerStarted","Data":"42883e4145243158a20d7c5367da4db36ec1e9a8c9f2425de0f8afb490df635f"}
Sep 19 08:20:39.490909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.490858    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:39.491096 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.490917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:39.491096 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.491051    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:39.491219 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.491117    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:43.491099183 +0000 UTC m=+11.575599987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:39.491535 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.491515    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:39.491600 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.491574    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:43.491558424 +0000 UTC m=+11.576059208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:39.692157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.692122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:39.692321 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.692277    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:39.692321 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.692297    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:39.692321 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.692311    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:39.692428 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.692366    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:20:43.69234781 +0000 UTC m=+11.776848595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:39.901263 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.901187    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:39.901423 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.901337    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:39.901737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.901714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:39.901832 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.901811    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:39.902007 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:39.901969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:39.902092 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:39.902076    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:41.904267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:41.904238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:41.904709 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:41.904363    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:41.904780 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:41.904718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:41.904850 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:41.904823    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:41.904963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:41.904870    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:41.904963 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:41.904934    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:43.524331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.524293    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:43.524824 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.524389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:43.524824 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.524458    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:43.524824 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.524506    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:43.524824 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.524535    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.524515474 +0000 UTC m=+19.609016270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:43.524824 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.524560    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.524543403 +0000 UTC m=+19.609044204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:43.725998 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.725943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:43.726154 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.726122    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:43.726154 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.726149    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:43.726263 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.726163    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:43.726263 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.726233    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.726213411 +0000 UTC m=+19.810714213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.902482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.902590    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.902934    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.903047    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:43.903097    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:43.903262 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:43.903179    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:45.904502 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:45.904293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:45.904943 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:45.904322    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:45.904943 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:45.904586    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:45.904943 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:45.904327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:45.904943 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:45.904666    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:45.904943 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:45.904784    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:47.905011 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:47.904964    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:47.905499 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:47.905079    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:47.905499 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:47.905089    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:47.905499 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:47.905113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:47.905499 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:47.905228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:47.905499 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:47.905311    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:49.901633 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:49.901599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:49.902088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:49.901641    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:49.902088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:49.901599    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:49.902088 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:49.901738    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:49.902088 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:49.901854    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:49.902088 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:49.901942    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:51.589713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.589670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:51.590192 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.589721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:51.590192 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.589802    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:51.590192 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.589844    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:51.590192 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.589875    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.589855714 +0000 UTC m=+35.674356509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:51.590192 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.589894    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.589883685 +0000 UTC m=+35.674384465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:51.791950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.791914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:51.792130 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.792059    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:51.792130 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.792077    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:51.792130 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.792087    2584 projected.go:196] Error preparing data for projected volume kube-api-access-znjn8 for pod openshift-network-diagnostics/network-check-target-2lpmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.792255 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.792133    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8 podName:55bd2504-c5a5-4c70-8425-aba0d52bf91e nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.792121112 +0000 UTC m=+35.876621893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-znjn8" (UniqueName: "kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8") pod "network-check-target-2lpmf" (UID: "55bd2504-c5a5-4c70-8425-aba0d52bf91e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.903547 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.903475    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:51.903716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.903582    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:51.903716 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.903591    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:51.903819 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.903717    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:51.903819 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:51.903770    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:51.903918 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:51.903842    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:53.901803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:53.901775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:53.902153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:53.901848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:53.902153 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:53.901868    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:53.902153 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:53.901946    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:53.902153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:53.902027    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:53.902153 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:53.902131    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:54.972343 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.972067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbq2l" event={"ID":"6171bdcd-0b75-4c34-8d84-9c28cfd810a0","Type":"ContainerStarted","Data":"7b82df95c16a4e22d7c1ac080462f168c78875cdd8d300a88a7eb1b5a18823c9"}
Sep 19 08:20:54.974049 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.974023    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="fc00923d9963c85ca91a3fd565e84124009a2e42530a90311763b8873e377e75" exitCode=0
Sep 19 08:20:54.974182 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.974106    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"fc00923d9963c85ca91a3fd565e84124009a2e42530a90311763b8873e377e75"}
Sep 19 08:20:54.976829 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.976790    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lmzgn" event={"ID":"028b379e-a3d7-4094-908c-e51d0ce4d1ff","Type":"ContainerStarted","Data":"c6ff27c359360df0502497ff06763fd23a8b6f4ec79e78431d9d6235de38d14d"}
Sep 19 08:20:54.980068 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980049    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:20:54.980464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980438    2584 generic.go:356] "Generic (PLEG): container finished" podID="5c2aa2ce-a894-4087-b089-83fa0a5a824c" containerID="5bdb4cb724918ae2409bc1926ab3dfe4d514342914689d8212bd1aeb3d0c59bc" exitCode=1
Sep 19 08:20:54.980565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980499    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"087f82067e431d2df71f29dc7e0be9ff9740914de4fee11ca9eb9ff44c61b501"}
Sep 19 08:20:54.980622 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980582    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"59c8b97855d3d1c2e3c62c4cc9f2da6135245561856f49902e1b0c1f2b390dd0"}
Sep 19 08:20:54.980671 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"0dfbff022e3cfde99dc94764b092d9d26edb52017c9010ba858177fdb2fbdb63"}
Sep 19 08:20:54.980720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980691    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"7f6c37a3e2080f7c43e6f25057d5dec53cc7c763bd00092e80e2edebe3e77862"}
Sep 19 08:20:54.980768 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerDied","Data":"5bdb4cb724918ae2409bc1926ab3dfe4d514342914689d8212bd1aeb3d0c59bc"}
Sep 19 08:20:54.980816 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.980786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"99386bfd62fb2bdf1c8a39181779f16b379cefe5fbbe75acb1e17f9c2d574deb"}
Sep 19 08:20:54.982313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.982287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w7k7" event={"ID":"bd450284-312e-40a7-9188-8a0692c0b872","Type":"ContainerStarted","Data":"da4d1e8d14e7c3f70a0e48deb1ac14c532a06baffea50bfb1a6d723177ff9b79"}
Sep 19 08:20:54.983910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.983887    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9h2kp" event={"ID":"3a2b6c98-8849-4a6f-87a6-648d889e4b4f","Type":"ContainerStarted","Data":"38afc2b4f4863c0fdd4e8a0d8a25e9efc55b76dc23b95bc1bb93accc37d28b84"}
Sep 19 08:20:54.985567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.985547    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474" event={"ID":"1be7dfc1-db2c-4a06-98e4-478250e634a9","Type":"ContainerStarted","Data":"feba440618b540f7dc4fb3739b2f3c6d072c2bfe21c68d909c0cec8844c88cf3"}
Sep 19 08:20:54.986299 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.986255    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-99.ec2.internal" podStartSLOduration=19.98623809 podStartE2EDuration="19.98623809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:38.963431208 +0000 UTC m=+7.047932012" watchObservedRunningTime="2026-09-19 08:20:54.98623809 +0000 UTC m=+23.070738895"
Sep 19 08:20:54.986425 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.986398    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbq2l" podStartSLOduration=7.928510621 podStartE2EDuration="20.986390247s" totalImagesPullingTime="13.057879626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:54.986008564 +0000 UTC m=+23.070509367" watchObservedRunningTime="2026-09-19 08:20:54.986390247 +0000 UTC m=+23.070891052"
Sep 19 08:20:54.987221 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.987200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2tqn" event={"ID":"0be6872e-0257-4d2a-8943-eec60b49bd88","Type":"ContainerStarted","Data":"92a6ee4b559893be148d7b420ca04bbb17967eb305758861b55480081b3065e7"}
Sep 19 08:20:54.998910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:54.998872    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lmzgn" podStartSLOduration=3.726978957 podStartE2EDuration="20.998860989s" totalImagesPullingTime="17.271882032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:54.998413103 +0000 UTC m=+23.082913906" watchObservedRunningTime="2026-09-19 08:20:54.998860989 +0000 UTC m=+23.083361794"
Sep 19 08:20:55.040131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.040090    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9h2kp" podStartSLOduration=3.73894663 podStartE2EDuration="21.04007614s" totalImagesPullingTime="17.30112951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.039693709 +0000 UTC m=+23.124194512" watchObservedRunningTime="2026-09-19 08:20:55.04007614 +0000 UTC m=+23.124576942"
Sep 19 08:20:55.056464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.056429    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5w7k7" podStartSLOduration=3.694739503 podStartE2EDuration="21.056418643s" totalImagesPullingTime="17.36167914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.056312171 +0000 UTC m=+23.140812975" watchObservedRunningTime="2026-09-19 08:20:55.056418643 +0000 UTC m=+23.140919446"
Sep 19 08:20:55.485145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.485011    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.904919    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.904955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.904960    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:55.905062    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:55.905163    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:55.905470 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:55.905244    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:55.928713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.928624    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:20:55.485141727Z","UUID":"02cdd1b2-9e9f-42a5-b914-eb9d78ee72e7","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:20:55.930300 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.930280    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:20:55.930440 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.930328    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:20:55.992565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.992533    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474" event={"ID":"1be7dfc1-db2c-4a06-98e4-478250e634a9","Type":"ContainerStarted","Data":"9b5368a0f65766aa0d9360f68621dcf68eb3d8434a426d4d9e62b136c76b4ccd"}
Sep 19 08:20:55.994729 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:55.994624    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgjpr" event={"ID":"b50d53d5-1085-4aea-8b4c-bcdcf003d4b5","Type":"ContainerStarted","Data":"b9c83632db4f908733ed1d06e7bdf60ba3aae635dc43e09d1d1dab287cc1fa18"}
Sep 19 08:20:56.009305 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:56.009263    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qgjpr" podStartSLOduration=4.704725141 podStartE2EDuration="22.009250598s" totalImagesPullingTime="17.304525457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:56.009110385 +0000 UTC m=+24.093611209" watchObservedRunningTime="2026-09-19 08:20:56.009250598 +0000 UTC m=+24.093751392"
Sep 19 08:20:56.009715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:56.009687    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p2tqn" podStartSLOduration=4.78908325 podStartE2EDuration="22.009678084s" totalImagesPullingTime="17.220594834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.081394683 +0000 UTC m=+23.165895499" watchObservedRunningTime="2026-09-19 08:20:56.009678084 +0000 UTC m=+24.094178887"
Sep 19 08:20:56.367864 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:56.367779    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:56.368390 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:56.368368    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:20:56.999442 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:56.999405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474" event={"ID":"1be7dfc1-db2c-4a06-98e4-478250e634a9","Type":"ContainerStarted","Data":"1585a9a8804437890651d5429074c7b30255aa799e554cbbcf3db6c0c392d741"}
Sep 19 08:20:57.003972 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.003932    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:20:57.004427 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.004405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"a6132484425e7bf5e8cf6ae4fd4a055ecc06e2e1363648a600da77cae362cc1d"}
Sep 19 08:20:57.018779 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.018724    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7474" podStartSLOduration=3.516225313 podStartE2EDuration="23.018705718s" totalImagesPullingTime="19.502480405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:57.017241831 +0000 UTC m=+25.101742644" watchObservedRunningTime="2026-09-19 08:20:57.018705718 +0000 UTC m=+25.103206523"
Sep 19 08:20:57.901061 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.901031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:57.901267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.901034    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:57.901267 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:57.901152    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:57.901267 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:57.901246    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:20:57.901267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:57.901034    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:57.901432 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:57.901356    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:58.006661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:58.006635    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:20:59.901378 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:59.901205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:20:59.901869 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:59.901215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:20:59.901869 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:59.901454    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:20:59.901869 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:20:59.901215    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:20:59.901869 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:59.901525    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:20:59.901869 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:20:59.901584    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:21:00.012906 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.012877    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="f9005ea0a7733ce65cb2a80cc2376f7e99a3a2aa38b8911b62e45aad665b982d" exitCode=0
Sep 19 08:21:00.013033 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.012969    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"f9005ea0a7733ce65cb2a80cc2376f7e99a3a2aa38b8911b62e45aad665b982d"}
Sep 19 08:21:00.021952 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.021929    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:21:00.022442 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.022415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"e3cc5dd5bc182ac9ba763dbbbbdf68502a031f8033492774dcac537f3f78fad0"}
Sep 19 08:21:00.022683 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.022668    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:00.022799 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.022783    2584 scope.go:117] "RemoveContainer" containerID="5bdb4cb724918ae2409bc1926ab3dfe4d514342914689d8212bd1aeb3d0c59bc"
Sep 19 08:21:00.040048 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:00.039952    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:01.032259 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.032234    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:21:01.032935 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.032907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" event={"ID":"5c2aa2ce-a894-4087-b089-83fa0a5a824c","Type":"ContainerStarted","Data":"416aebc3fe377af7cd5ebbb31ed1edcd04a381da77123eaa5f8638336ae29d6d"}
Sep 19 08:21:01.033095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.033017    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:01.033323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.033297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:01.050465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.050433    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:01.061609 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.061573    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" podStartSLOduration=9.743627453 podStartE2EDuration="27.061555965s" totalImagesPullingTime="17.317928512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:01.05964847 +0000 UTC m=+29.144149272" watchObservedRunningTime="2026-09-19 08:21:01.061555965 +0000 UTC m=+29.146056771"
Sep 19 08:21:01.299355 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.299300    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lpmf"]
Sep 19 08:21:01.299463 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.299413    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:01.299538 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:01.299517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:21:01.302344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.302321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6tqd"]
Sep 19 08:21:01.302464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.302425    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:01.302542 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:01.302521    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:21:01.302962 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.302937    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s4btv"]
Sep 19 08:21:01.303072 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:01.303058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:01.303184 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:01.303164    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:21:02.036854 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.036814    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="3c5fe4131370e1e2dbc14ba6167209e217abe7f5120276d189a3709c6209c71c" exitCode=0
Sep 19 08:21:02.037288 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.036901    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"3c5fe4131370e1e2dbc14ba6167209e217abe7f5120276d189a3709c6209c71c"}
Sep 19 08:21:02.037288 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.036998    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:02.901634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.901607    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:02.901777 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.901697    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:02.901777 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:02.901702    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:21:02.901777 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:02.901724    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:02.901879 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:02.901800    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:21:02.901913 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:02.901881    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:21:03.039001 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:03.038965    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:03.937136 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:03.936966    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:04.043341 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.043312    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="1881af7445c679240c6ba7141e3384415c488609dbdaefe1d58845d785445d10" exitCode=0
Sep 19 08:21:04.043626 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.043396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"1881af7445c679240c6ba7141e3384415c488609dbdaefe1d58845d785445d10"}
Sep 19 08:21:04.055996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.055947    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch" podUID="5c2aa2ce-a894-4087-b089-83fa0a5a824c" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 19 08:21:04.900859 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.900823    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:04.901067 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.900945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:04.901067 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:04.900959    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:04.901067 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:04.900951    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:21:04.901241 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:04.901082    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:21:04.901241 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:04.901221    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:21:05.139346 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:05.139233    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:21:05.139763 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:05.139368    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:05.139954 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:05.139935    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbq2l"
Sep 19 08:21:06.900967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:06.900935    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:06.901415 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:06.900945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:06.901415 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:06.901060    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6tqd" podUID="c4775e37-d3b3-414d-a623-177de1acbfc0"
Sep 19 08:21:06.901415 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:06.901084    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:06.901415 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:06.901155    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:21:06.901415 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:06.901242    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lpmf" podUID="55bd2504-c5a5-4c70-8425-aba0d52bf91e"
Sep 19 08:21:07.194295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.194251    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-99.ec2.internal" event="NodeReady"
Sep 19 08:21:07.194453 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.194426    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:07.242090 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.242052    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"]
Sep 19 08:21:07.271266 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.271243    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6778975687-jmfgj"]
Sep 19 08:21:07.271420 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.271370    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"
Sep 19 08:21:07.274739 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.274717    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:07.274873 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.274717    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t6bkm\""
Sep 19 08:21:07.275041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.275024    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:07.293660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.293612    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"]
Sep 19 08:21:07.293660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.293620    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.295959 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.295941    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 08:21:07.296088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.295976    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 08:21:07.296088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.296068    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 08:21:07.296204 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.296068    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lv4hf\""
Sep 19 08:21:07.302200 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.302184    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 08:21:07.312773 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.312754    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"]
Sep 19 08:21:07.312860 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.312782    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4vz85"]
Sep 19 08:21:07.312921 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.312890    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.315552 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.315528    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 19 08:21:07.315670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.315574    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jbjhg\""
Sep 19 08:21:07.316094 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.316065    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 19 08:21:07.327135 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.327117    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6778975687-jmfgj"]
Sep 19 08:21:07.327234 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.327144    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlcm"]
Sep 19 08:21:07.327288 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.327240    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.329772 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.329756    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:07.330099 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.330082    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dzp9\""
Sep 19 08:21:07.330326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.330311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:07.347603 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.347583    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"]
Sep 19 08:21:07.347695 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.347617    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vz85"]
Sep 19 08:21:07.347695 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.347642    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlcm"]
Sep 19 08:21:07.347818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.347706    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.351258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.351235    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:07.351370 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.351270    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:07.351555 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.351534    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:07.351641 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.351570    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2s5wr\""
Sep 19 08:21:07.415781 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415753    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.415897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415819    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.415897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.415897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415894    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.416077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.415933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.416026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvv9\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.416070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.416097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.416131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.416241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.416163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8sh\" (UniqueName: \"kubernetes.io/projected/53dccffe-580e-4996-ad9d-0bb39cbeb80f-kube-api-access-rw8sh\") pod \"network-check-source-7cbc45fddc-c5sfz\" (UID: \"53dccffe-580e-4996-ad9d-0bb39cbeb80f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"
Sep 19 08:21:07.517094 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twr5c\" (UniqueName: \"kubernetes.io/projected/abb7958d-0540-407c-ba3e-f9d28f8115af-kube-api-access-twr5c\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8sh\" (UniqueName: \"kubernetes.io/projected/53dccffe-580e-4996-ad9d-0bb39cbeb80f-kube-api-access-rw8sh\") pod \"network-check-source-7cbc45fddc-c5sfz\" (UID: \"53dccffe-580e-4996-ad9d-0bb39cbeb80f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517178    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517215    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.517234    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.517249 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.517251    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da27a61-af20-4407-934d-eda26bccb1b6-config-volume\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8m6g\" (UniqueName: \"kubernetes.io/projected/6da27a61-af20-4407-934d-eda26bccb1b6-kube-api-access-p8m6g\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.517311    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.017291252 +0000 UTC m=+36.101792050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.517367    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.517423    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.017408916 +0000 UTC m=+36.101909698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.517575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.518024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da27a61-af20-4407-934d-eda26bccb1b6-tmp-dir\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.518024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvv9\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.518024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb7958d-0540-407c-ba3e-f9d28f8115af-cert\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.518024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.518024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.517749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.518281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.518209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.518335 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.518310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.518391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.518336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:07.518391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.518376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.521727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.521701    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.521727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.521715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.529509 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.529463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8sh\" (UniqueName: \"kubernetes.io/projected/53dccffe-580e-4996-ad9d-0bb39cbeb80f-kube-api-access-rw8sh\") pod \"network-check-source-7cbc45fddc-c5sfz\" (UID: \"53dccffe-580e-4996-ad9d-0bb39cbeb80f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"
Sep 19 08:21:07.530364 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.530344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvv9\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.530587 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.530567    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:07.583421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.583364    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"
Sep 19 08:21:07.618186 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twr5c\" (UniqueName: \"kubernetes.io/projected/abb7958d-0540-407c-ba3e-f9d28f8115af-kube-api-access-twr5c\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.618273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:07.618319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da27a61-af20-4407-934d-eda26bccb1b6-config-volume\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.618319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8m6g\" (UniqueName: \"kubernetes.io/projected/6da27a61-af20-4407-934d-eda26bccb1b6-kube-api-access-p8m6g\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.618381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:07.618381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.618381 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618361    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:07.618381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da27a61-af20-4407-934d-eda26bccb1b6-tmp-dir\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.618497 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618410    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret podName:c4775e37-d3b3-414d-a623-177de1acbfc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.618397797 +0000 UTC m=+67.702898578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret") pod "global-pull-secret-syncer-j6tqd" (UID: "c4775e37-d3b3-414d-a623-177de1acbfc0") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:07.618497 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb7958d-0540-407c-ba3e-f9d28f8115af-cert\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.618667 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.618645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6da27a61-af20-4407-934d-eda26bccb1b6-tmp-dir\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.618754 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618739    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:07.618827 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618777    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.618767064 +0000 UTC m=+67.703267845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:07.618887 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618833    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:07.618939 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:07.618888    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.118870209 +0000 UTC m=+36.203371005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:07.619105 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.619082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6da27a61-af20-4407-934d-eda26bccb1b6-config-volume\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.620699 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.620678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb7958d-0540-407c-ba3e-f9d28f8115af-cert\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.626518 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.626494    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8m6g\" (UniqueName: \"kubernetes.io/projected/6da27a61-af20-4407-934d-eda26bccb1b6-kube-api-access-p8m6g\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:07.626949 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.626922    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twr5c\" (UniqueName: \"kubernetes.io/projected/abb7958d-0540-407c-ba3e-f9d28f8115af-kube-api-access-twr5c\") pod \"ingress-canary-pmlcm\" (UID: \"abb7958d-0540-407c-ba3e-f9d28f8115af\") " pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.657178 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.657153    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmlcm"
Sep 19 08:21:07.819532 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.819491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:07.822429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:07.822397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjn8\" (UniqueName: \"kubernetes.io/projected/55bd2504-c5a5-4c70-8425-aba0d52bf91e-kube-api-access-znjn8\") pod \"network-check-target-2lpmf\" (UID: \"55bd2504-c5a5-4c70-8425-aba0d52bf91e\") " pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:08.022110 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.022078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.022133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.022257    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.022278    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.022344    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.02232536 +0000 UTC m=+37.106826158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.022357    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:08.022606 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.022412    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.022398819 +0000 UTC m=+37.106899615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:08.123252 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.123220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:08.123426 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.123396    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.123491 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:08.123474    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.123453845 +0000 UTC m=+37.207954645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.368480 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.368260    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz"]
Sep 19 08:21:08.369039 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.369022    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmlcm"]
Sep 19 08:21:08.377666 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:21:08.377642    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dccffe_580e_4996_ad9d_0bb39cbeb80f.slice/crio-21806e32e2de20ae945549bd2a11f0dbb4dbb4ec830e56d0292640f2e4920eca WatchSource:0}: Error finding container 21806e32e2de20ae945549bd2a11f0dbb4dbb4ec830e56d0292640f2e4920eca: Status 404 returned error can't find the container with id 21806e32e2de20ae945549bd2a11f0dbb4dbb4ec830e56d0292640f2e4920eca
Sep 19 08:21:08.378052 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:21:08.378037    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb7958d_0540_407c_ba3e_f9d28f8115af.slice/crio-2208ef476212525785e48c57c6289a55ccac9bc3d14c01199e571d9ae6423697 WatchSource:0}: Error finding container 2208ef476212525785e48c57c6289a55ccac9bc3d14c01199e571d9ae6423697: Status 404 returned error can't find the container with id 2208ef476212525785e48c57c6289a55ccac9bc3d14c01199e571d9ae6423697
Sep 19 08:21:08.901454 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.901409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:08.901712 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.901417    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:08.901712 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.901542    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:08.904102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.904079    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:08.905152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.904973    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:08.905152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.905020    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95gj6\""
Sep 19 08:21:08.905152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.905026    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5xtcl\""
Sep 19 08:21:08.912328 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:08.912309    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:09.030574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.030544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.030622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.030658    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.030676    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.030737    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:11.030716509 +0000 UTC m=+39.115217306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.030784    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:09.031350 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.030845    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:11.0308289 +0000 UTC m=+39.115329699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:09.059477 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.059244    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="507787e5122cda4c009ddc512687f123b30e721cdcf3d54e2711878a74487e29" exitCode=0
Sep 19 08:21:09.059650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.059480    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"507787e5122cda4c009ddc512687f123b30e721cdcf3d54e2711878a74487e29"}
Sep 19 08:21:09.061384 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.061276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmlcm" event={"ID":"abb7958d-0540-407c-ba3e-f9d28f8115af","Type":"ContainerStarted","Data":"2208ef476212525785e48c57c6289a55ccac9bc3d14c01199e571d9ae6423697"}
Sep 19 08:21:09.064104 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.064076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz" event={"ID":"53dccffe-580e-4996-ad9d-0bb39cbeb80f","Type":"ContainerStarted","Data":"21806e32e2de20ae945549bd2a11f0dbb4dbb4ec830e56d0292640f2e4920eca"}
Sep 19 08:21:09.131910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.131402    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:09.131910 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.131541    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:09.131910 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:09.131596    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:11.131578305 +0000 UTC m=+39.216079092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:09.204122 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:09.203920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lpmf"]
Sep 19 08:21:09.207579 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:21:09.207416    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bd2504_c5a5_4c70_8425_aba0d52bf91e.slice/crio-8daf6200ecab9243a550f6ad6ff643b20487dc446835a7347740ee6869f05240 WatchSource:0}: Error finding container 8daf6200ecab9243a550f6ad6ff643b20487dc446835a7347740ee6869f05240: Status 404 returned error can't find the container with id 8daf6200ecab9243a550f6ad6ff643b20487dc446835a7347740ee6869f05240
Sep 19 08:21:10.072941 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:10.072905    2584 generic.go:356] "Generic (PLEG): container finished" podID="9920c568-66e7-43c9-839d-8b8846d5d8c2" containerID="2bc0147b5c26e3efef2c80efb0263322cdbfb97b2b49558c2b16bfe2eb485649" exitCode=0
Sep 19 08:21:10.073510 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:10.072998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerDied","Data":"2bc0147b5c26e3efef2c80efb0263322cdbfb97b2b49558c2b16bfe2eb485649"}
Sep 19 08:21:10.075155 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:10.075114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lpmf" event={"ID":"55bd2504-c5a5-4c70-8425-aba0d52bf91e","Type":"ContainerStarted","Data":"8daf6200ecab9243a550f6ad6ff643b20487dc446835a7347740ee6869f05240"}
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:11.046518    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.046644    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.046945    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.047028    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.047006773 +0000 UTC m=+43.131507575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:11.047059    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.047152    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:11.047222 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.047212    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.047193752 +0000 UTC m=+43.131694535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:11.082817 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:11.082782    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r62dt" event={"ID":"9920c568-66e7-43c9-839d-8b8846d5d8c2","Type":"ContainerStarted","Data":"2f0d90033665d17571108c40ef21d4f220fd030ffa1fce93c3234a2c23d6c60e"}
Sep 19 08:21:11.104861 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:11.104706    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r62dt" podStartSLOduration=9.187027354 podStartE2EDuration="37.104687509s" totalImagesPullingTime="27.917660155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.103371337 +0000 UTC m=+39.187872141" watchObservedRunningTime="2026-09-19 08:21:11.104687509 +0000 UTC m=+39.189188313"
Sep 19 08:21:11.148937 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:11.148483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:11.148937 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.148585    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.148937 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:11.148640    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.148621879 +0000 UTC m=+43.233122667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:12.090379 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.090286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz" event={"ID":"53dccffe-580e-4996-ad9d-0bb39cbeb80f","Type":"ContainerStarted","Data":"143201d054f223c9776a20823437ae8ff05e6829dfca69ba46e86ac6d7e53a6a"}
Sep 19 08:21:12.091953 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.091928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lpmf" event={"ID":"55bd2504-c5a5-4c70-8425-aba0d52bf91e","Type":"ContainerStarted","Data":"39628a6fc12a91300641e77fb4a4b6b24d9f8853c5e90107020ae8d5c825a99f"}
Sep 19 08:21:12.092105 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.091976    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:12.093586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.093560    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmlcm" event={"ID":"abb7958d-0540-407c-ba3e-f9d28f8115af","Type":"ContainerStarted","Data":"cde10d4d6511611d1e908042897a7c842942694bdb6f2d935cca490db0d24cd4"}
Sep 19 08:21:12.107144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.107106    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-c5sfz" podStartSLOduration=33.617486265 podStartE2EDuration="37.107093269s" totalImagesPullingTime="3.489607004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.106946878 +0000 UTC m=+40.191447682" watchObservedRunningTime="2026-09-19 08:21:12.107093269 +0000 UTC m=+40.191594089"
Sep 19 08:21:12.121940 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.121901    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2lpmf" podStartSLOduration=35.457026238 podStartE2EDuration="38.121888675s" totalImagesPullingTime="2.664862437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.121493807 +0000 UTC m=+40.205994611" watchObservedRunningTime="2026-09-19 08:21:12.121888675 +0000 UTC m=+40.206389484"
Sep 19 08:21:12.138053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:12.138017    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pmlcm" podStartSLOduration=1.68301525 podStartE2EDuration="5.138007091s" totalImagesPullingTime="3.454991841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.137537334 +0000 UTC m=+40.222038137" watchObservedRunningTime="2026-09-19 08:21:12.138007091 +0000 UTC m=+40.222507894"
Sep 19 08:21:15.077220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:15.077189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:15.077258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.077338    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.077366    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.077377    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.077408    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.07739227 +0000 UTC m=+51.161893052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:15.077574 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.077423    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.077416608 +0000 UTC m=+51.161917388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:15.177907 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:15.177882    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:15.178048 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.177977    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:15.178048 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:15.178036    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.178020896 +0000 UTC m=+51.262521696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.134951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:23.134912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:23.134996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.135123    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.135138    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.135150    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.135191    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.135170952 +0000 UTC m=+67.219671736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:23.135338 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.135212    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.135200286 +0000 UTC m=+67.219701066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:23.238958 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:23.236226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:23.238958 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.236478    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.238958 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:23.236599    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.236580163 +0000 UTC m=+67.321080958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:34.052763 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:34.052740    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sbhch"
Sep 19 08:21:39.148785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.148751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.148809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.148920    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.148928    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.148939    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.149030    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:11.149015045 +0000 UTC m=+99.233515826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:39.149291 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.149045    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:11.149038791 +0000 UTC m=+99.233539572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:21:39.249312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.249277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:21:39.249445 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.249408    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:39.249497 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.249459    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:11.249444772 +0000 UTC m=+99.333945556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:39.652796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.652754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:21:39.652948 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.652921    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:39.655258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.655234    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:39.655380 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.655312    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:39.663240 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.663219    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:21:39.663324 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:21:39.663281    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:43.663261896 +0000 UTC m=+131.747762697 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : secret "metrics-daemon-secret" not found
Sep 19 08:21:39.666350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.666334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4775e37-d3b3-414d-a623-177de1acbfc0-original-pull-secret\") pod \"global-pull-secret-syncer-j6tqd\" (UID: \"c4775e37-d3b3-414d-a623-177de1acbfc0\") " pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:39.825733 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:39.825712    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6tqd"
Sep 19 08:21:40.090342 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:40.090316    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6tqd"]
Sep 19 08:21:40.093327 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:21:40.093295    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4775e37_d3b3_414d_a623_177de1acbfc0.slice/crio-840815c40d3535cc6b212e1e56b5617db9f1399fc3f393a31bfc4e0a619afb5f WatchSource:0}: Error finding container 840815c40d3535cc6b212e1e56b5617db9f1399fc3f393a31bfc4e0a619afb5f: Status 404 returned error can't find the container with id 840815c40d3535cc6b212e1e56b5617db9f1399fc3f393a31bfc4e0a619afb5f
Sep 19 08:21:40.165440 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:40.165414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6tqd" event={"ID":"c4775e37-d3b3-414d-a623-177de1acbfc0","Type":"ContainerStarted","Data":"840815c40d3535cc6b212e1e56b5617db9f1399fc3f393a31bfc4e0a619afb5f"}
Sep 19 08:21:43.099102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:43.099075    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2lpmf"
Sep 19 08:21:44.177739 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:21:44.177706    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6tqd" event={"ID":"c4775e37-d3b3-414d-a623-177de1acbfc0","Type":"ContainerStarted","Data":"1b4fa63562329620934e8b79ac5d5e45a981a0631bcf1fc00b4abacd882bd5ae"}
Sep 19 08:22:11.176496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:22:11.176452    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:22:11.176496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:22:11.176508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:22:11.177040 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.176633    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:22:11.177040 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.176645    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:22:11.177040 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.176669    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:22:11.177040 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.176687    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:23:15.176672093 +0000 UTC m=+163.261172875 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:22:11.177040 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.176741    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:23:15.176721495 +0000 UTC m=+163.261222303 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:22:11.277441 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:22:11.277414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:22:11.277557 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.277509    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:22:11.277557 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:11.277553    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:23:15.27754031 +0000 UTC m=+163.362041090 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:22:43.691681 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:22:43.691641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:22:43.692255 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:43.691797    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:22:43.692255 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:22:43.691888    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:24:45.691867793 +0000 UTC m=+253.776368599 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : secret "metrics-daemon-secret" not found
Sep 19 08:23:10.303820 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:10.303773    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6778975687-jmfgj" podUID="93d30278-e75d-42cd-b133-63d05821ab87"
Sep 19 08:23:10.322903 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:10.322882    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" podUID="8e683a50-2d02-4b9b-b0cb-8744e2790ef9"
Sep 19 08:23:10.336069 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:10.336050    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vz85" podUID="6da27a61-af20-4407-934d-eda26bccb1b6"
Sep 19 08:23:10.386146 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:10.386122    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:23:10.386251 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:10.386129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:23:10.386313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:10.386129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:23:11.920046 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:11.920000    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:23:15.184619 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:15.184574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:15.184643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.184721    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.184732    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.184745    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.184795    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:17.184778652 +0000 UTC m=+285.269279432 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:23:15.185024 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.184807    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:17.184800722 +0000 UTC m=+285.269301502 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:23:15.285608 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:15.285580    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:23:15.285732 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.285683    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:23:15.285732 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:23:15.285723    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:17.285710891 +0000 UTC m=+285.370211673 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:23:24.901211 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:23:24.901131    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:24:45.772914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:24:45.772818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:24:45.773438 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:24:45.772978    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:24:45.773438 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:24:45.773062    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs podName:d0b18af0-056d-42a2-aee2-e0fb3d43a166 nodeName:}" failed. No retries permitted until 2026-09-19 08:26:47.773045198 +0000 UTC m=+375.857545984 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs") pod "network-metrics-daemon-s4btv" (UID: "d0b18af0-056d-42a2-aee2-e0fb3d43a166") : secret "metrics-daemon-secret" not found
Sep 19 08:25:03.450875 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.450812    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j6tqd" podStartSLOduration=265.899475512 podStartE2EDuration="4m29.450795352s" totalImagesPullingTime="3.55131984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:44.206992989 +0000 UTC m=+72.291493783" watchObservedRunningTime="2026-09-19 08:25:03.450795352 +0000 UTC m=+271.535296155"
Sep 19 08:25:03.451421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.451090    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"]
Sep 19 08:25:03.453280 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.453253    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.454485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.454423    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-6sw7w"]
Sep 19 08:25:03.456406 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.456383    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.458271 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458240    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 19 08:25:03.458410 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458275    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 19 08:25:03.458410 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458330    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.458601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458416    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fl5gq\""
Sep 19 08:25:03.458875 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458839    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 19 08:25:03.458975 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.458955    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.459170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.459136    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.459287 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.459275    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hj2x9\""
Sep 19 08:25:03.459568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.459547    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 19 08:25:03.459850 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.459826    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mxmrg"]
Sep 19 08:25:03.462103 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.462082    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9b7b7b666-vbwq7"]
Sep 19 08:25:03.462202 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.462192    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.463496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.463471    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.464191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.464170    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.464557 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.464534    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 19 08:25:03.464995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.464948    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wvzrl\""
Sep 19 08:25:03.468356 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.468332    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 19 08:25:03.468356 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.468358    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.469152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.469066    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 19 08:25:03.469720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.469691    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.470238 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.470215    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 19 08:25:03.470400 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.470224    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 19 08:25:03.470947 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.470927    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 19 08:25:03.471665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.471643    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.472315 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.472285    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z9qkd\""
Sep 19 08:25:03.473603 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.473581    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.476430 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.476402    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 19 08:25:03.476541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.476523    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"]
Sep 19 08:25:03.478738 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.478712    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-6sw7w"]
Sep 19 08:25:03.479115 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.479094    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 19 08:25:03.479724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.479700    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9b7b7b666-vbwq7"]
Sep 19 08:25:03.497357 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.497337    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mxmrg"]
Sep 19 08:25:03.553698 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.553673    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"]
Sep 19 08:25:03.555456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.555439    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"]
Sep 19 08:25:03.555586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.555555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.557331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.557315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.558818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.558799    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-s72jl\""
Sep 19 08:25:03.558914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.558836    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 19 08:25:03.559082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.559066    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 19 08:25:03.559157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.559111    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.559157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.559117    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.559585 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.559571    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qx7j9\""
Sep 19 08:25:03.559923 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.559910    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 19 08:25:03.560059 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.560045    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:25:03.560279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.560266    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 19 08:25:03.574214 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.574194    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"]
Sep 19 08:25:03.575130 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.575115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"]
Sep 19 08:25:03.591600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591581    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-stats-auth\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.591694 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591607    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.591694 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591623    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-trusted-ca\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.591694 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97a94cfc-296d-4617-9970-fc8d7620c8b0-serving-cert\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.591818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-default-certificate\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.591818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-config\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.591818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591777    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2xg\" (UniqueName: \"kubernetes.io/projected/5040f0db-be61-4496-b978-27715767d629-kube-api-access-xn2xg\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591825    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848kv\" (UniqueName: \"kubernetes.io/projected/97a94cfc-296d-4617-9970-fc8d7620c8b0-kube-api-access-848kv\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-snapshots\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.591928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040f0db-be61-4496-b978-27715767d629-serving-cert\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.592185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.591973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.592185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.592003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-tmp\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.592185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.592029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccscb\" (UniqueName: \"kubernetes.io/projected/1211f7b0-61e1-4389-854d-0fef63b6c75f-kube-api-access-ccscb\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.592185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.592056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pfv\" (UniqueName: \"kubernetes.io/projected/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-kube-api-access-r9pfv\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.592185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.592072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693183 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693160    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97a94cfc-296d-4617-9970-fc8d7620c8b0-serving-cert\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693189    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.693273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-default-certificate\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.693273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-config\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.693273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2xg\" (UniqueName: \"kubernetes.io/projected/5040f0db-be61-4496-b978-27715767d629-kube-api-access-xn2xg\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.693431 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848kv\" (UniqueName: \"kubernetes.io/projected/97a94cfc-296d-4617-9970-fc8d7620c8b0-kube-api-access-848kv\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693431 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.693431 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.693420    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:25:03.693549 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.693482    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:04.193464558 +0000 UTC m=+272.277965359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : secret "router-metrics-certs-default" not found
Sep 19 08:25:03.693549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.693549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjv6\" (UniqueName: \"kubernetes.io/projected/7b3384b3-a1de-4edf-898e-dc3a77db41bf-kube-api-access-rcjv6\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.693706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-snapshots\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693646    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b3384b3-a1de-4edf-898e-dc3a77db41bf-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.693706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693678    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040f0db-be61-4496-b978-27715767d629-serving-cert\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-tmp\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccscb\" (UniqueName: \"kubernetes.io/projected/1211f7b0-61e1-4389-854d-0fef63b6c75f-kube-api-access-ccscb\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pfv\" (UniqueName: \"kubernetes.io/projected/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-kube-api-access-r9pfv\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b3384b3-a1de-4edf-898e-dc3a77db41bf-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.693900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693884    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-stats-auth\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693904    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-trusted-ca\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.693946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckc7\" (UniqueName: \"kubernetes.io/projected/3ffa901d-4b87-410b-96e3-e02b97be8fda-kube-api-access-4ckc7\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.694142    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.694190    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:03.694273 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.694273    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:04.19425619 +0000 UTC m=+272.278756976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:03.694559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.694285    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-config\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.694559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.694389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.694559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.694414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-tmp\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.694559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.694444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/97a94cfc-296d-4617-9970-fc8d7620c8b0-snapshots\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.694762 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.694747    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:04.194729508 +0000 UTC m=+272.279230307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:03.695098 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.695074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5040f0db-be61-4496-b978-27715767d629-trusted-ca\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.695226 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.695206    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97a94cfc-296d-4617-9970-fc8d7620c8b0-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.695996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.695959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-default-certificate\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.696475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.696454    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97a94cfc-296d-4617-9970-fc8d7620c8b0-serving-cert\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.696615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.696595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040f0db-be61-4496-b978-27715767d629-serving-cert\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.696936 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.696921    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-stats-auth\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.701644 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.701585    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pfv\" (UniqueName: \"kubernetes.io/projected/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-kube-api-access-r9pfv\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:03.701856 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.701839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2xg\" (UniqueName: \"kubernetes.io/projected/5040f0db-be61-4496-b978-27715767d629-kube-api-access-xn2xg\") pod \"console-operator-fd4bb46c-6sw7w\" (UID: \"5040f0db-be61-4496-b978-27715767d629\") " pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.702796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.702762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-848kv\" (UniqueName: \"kubernetes.io/projected/97a94cfc-296d-4617-9970-fc8d7620c8b0-kube-api-access-848kv\") pod \"insights-operator-6f9c644dd9-mxmrg\" (UID: \"97a94cfc-296d-4617-9970-fc8d7620c8b0\") " pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.702907 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.702888    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccscb\" (UniqueName: \"kubernetes.io/projected/1211f7b0-61e1-4389-854d-0fef63b6c75f-kube-api-access-ccscb\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:03.773971 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.773954    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:03.788604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.788585    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg"
Sep 19 08:25:03.794646 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.794623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjv6\" (UniqueName: \"kubernetes.io/projected/7b3384b3-a1de-4edf-898e-dc3a77db41bf-kube-api-access-rcjv6\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.794716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.794665    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b3384b3-a1de-4edf-898e-dc3a77db41bf-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.794765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.794734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b3384b3-a1de-4edf-898e-dc3a77db41bf-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.794813 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.794787    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckc7\" (UniqueName: \"kubernetes.io/projected/3ffa901d-4b87-410b-96e3-e02b97be8fda-kube-api-access-4ckc7\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.794866 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.794841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.794968 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.794946    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:25:03.795120 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:03.795033    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls podName:3ffa901d-4b87-410b-96e3-e02b97be8fda nodeName:}" failed. No retries permitted until 2026-09-19 08:25:04.295015935 +0000 UTC m=+272.379516729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z5jq2" (UID: "3ffa901d-4b87-410b-96e3-e02b97be8fda") : secret "samples-operator-tls" not found
Sep 19 08:25:03.795372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.795352    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b3384b3-a1de-4edf-898e-dc3a77db41bf-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.797017 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.796973    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b3384b3-a1de-4edf-898e-dc3a77db41bf-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.802772 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.802750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckc7\" (UniqueName: \"kubernetes.io/projected/3ffa901d-4b87-410b-96e3-e02b97be8fda-kube-api-access-4ckc7\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:03.802873 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.802798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjv6\" (UniqueName: \"kubernetes.io/projected/7b3384b3-a1de-4edf-898e-dc3a77db41bf-kube-api-access-rcjv6\") pod \"kube-storage-version-migrator-operator-55889f7b47-jx5qr\" (UID: \"7b3384b3-a1de-4edf-898e-dc3a77db41bf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:03.865634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:03.865611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"
Sep 19 08:25:04.042015 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.041974    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-6sw7w"]
Sep 19 08:25:04.044735 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:04.044711    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5040f0db_be61_4496_b978_27715767d629.slice/crio-c927e16a861a4d550c8eb0dcddb20aae6644867f255707750a384bfd84c7fff3 WatchSource:0}: Error finding container c927e16a861a4d550c8eb0dcddb20aae6644867f255707750a384bfd84c7fff3: Status 404 returned error can't find the container with id c927e16a861a4d550c8eb0dcddb20aae6644867f255707750a384bfd84c7fff3
Sep 19 08:25:04.064257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.064236    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mxmrg"]
Sep 19 08:25:04.066791 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:04.066773    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a94cfc_296d_4617_9970_fc8d7620c8b0.slice/crio-ff2d4421675db0998fb97a8dec9749d413e84145b10b7ddf845c037af8373ed2 WatchSource:0}: Error finding container ff2d4421675db0998fb97a8dec9749d413e84145b10b7ddf845c037af8373ed2: Status 404 returned error can't find the container with id ff2d4421675db0998fb97a8dec9749d413e84145b10b7ddf845c037af8373ed2
Sep 19 08:25:04.130143 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.130115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr"]
Sep 19 08:25:04.133539 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:04.133519    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3384b3_a1de_4edf_898e_dc3a77db41bf.slice/crio-b10bcb136e74b696c721bd9891d518fc66885e380f9177408adafc9ebea91665 WatchSource:0}: Error finding container b10bcb136e74b696c721bd9891d518fc66885e380f9177408adafc9ebea91665: Status 404 returned error can't find the container with id b10bcb136e74b696c721bd9891d518fc66885e380f9177408adafc9ebea91665
Sep 19 08:25:04.198075 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.198053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:04.198147 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.198123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:04.198195 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.198157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:04.198233 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.198209    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:05.198191863 +0000 UTC m=+273.282692643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:04.198324 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.198248    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:04.198324 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.198249    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:25:04.198324 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.198294    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:05.198281769 +0000 UTC m=+273.282782550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : secret "router-metrics-certs-default" not found
Sep 19 08:25:04.198324 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.198307    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:05.198300814 +0000 UTC m=+273.282801595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:04.299146 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.299095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:04.299227 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.299201    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:25:04.299264 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:04.299258    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls podName:3ffa901d-4b87-410b-96e3-e02b97be8fda nodeName:}" failed. No retries permitted until 2026-09-19 08:25:05.299246086 +0000 UTC m=+273.383746866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z5jq2" (UID: "3ffa901d-4b87-410b-96e3-e02b97be8fda") : secret "samples-operator-tls" not found
Sep 19 08:25:04.660247 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.660169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" event={"ID":"5040f0db-be61-4496-b978-27715767d629","Type":"ContainerStarted","Data":"c927e16a861a4d550c8eb0dcddb20aae6644867f255707750a384bfd84c7fff3"}
Sep 19 08:25:04.661424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.661392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr" event={"ID":"7b3384b3-a1de-4edf-898e-dc3a77db41bf","Type":"ContainerStarted","Data":"b10bcb136e74b696c721bd9891d518fc66885e380f9177408adafc9ebea91665"}
Sep 19 08:25:04.662543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:04.662518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg" event={"ID":"97a94cfc-296d-4617-9970-fc8d7620c8b0","Type":"ContainerStarted","Data":"ff2d4421675db0998fb97a8dec9749d413e84145b10b7ddf845c037af8373ed2"}
Sep 19 08:25:05.207964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:05.207678    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:05.207964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:05.207765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:05.207964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:05.207844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:05.207964 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.207937    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:05.208300 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.208035    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:07.208015595 +0000 UTC m=+275.292516390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:05.208300 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.208062    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:07.208042436 +0000 UTC m=+275.292543218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:05.208300 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.207942    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:25:05.208300 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.208173    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:07.208153156 +0000 UTC m=+275.292653942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : secret "router-metrics-certs-default" not found
Sep 19 08:25:05.308997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:05.308952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:05.309166 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.309111    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:25:05.309229 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:05.309179    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls podName:3ffa901d-4b87-410b-96e3-e02b97be8fda nodeName:}" failed. No retries permitted until 2026-09-19 08:25:07.309162794 +0000 UTC m=+275.393663588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z5jq2" (UID: "3ffa901d-4b87-410b-96e3-e02b97be8fda") : secret "samples-operator-tls" not found
Sep 19 08:25:07.225014 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.224969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.225086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.225133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.225153    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:11.225129881 +0000 UTC m=+279.309630688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.225225    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.225234    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.225280    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:11.225264542 +0000 UTC m=+279.309765329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:07.225357 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.225297    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:11.225287723 +0000 UTC m=+279.309788511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : secret "router-metrics-certs-default" not found
Sep 19 08:25:07.325457 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.325429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:07.325577 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.325542    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:25:07.325646 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:07.325591    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls podName:3ffa901d-4b87-410b-96e3-e02b97be8fda nodeName:}" failed. No retries permitted until 2026-09-19 08:25:11.325576502 +0000 UTC m=+279.410077285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z5jq2" (UID: "3ffa901d-4b87-410b-96e3-e02b97be8fda") : secret "samples-operator-tls" not found
Sep 19 08:25:07.673290 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.673224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg" event={"ID":"97a94cfc-296d-4617-9970-fc8d7620c8b0","Type":"ContainerStarted","Data":"f1138d02c80cab2b5872ccf617b20798965f74dd09cdfc35ceb6d4ce4fc507fb"}
Sep 19 08:25:07.675080 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.675061    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/0.log"
Sep 19 08:25:07.675153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.675103    2584 generic.go:356] "Generic (PLEG): container finished" podID="5040f0db-be61-4496-b978-27715767d629" containerID="8f12908e2afa8a57ae2b9d5ea39e24c2940945a5ef7276952cdc7ccc02740ec4" exitCode=255
Sep 19 08:25:07.675207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.675180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" event={"ID":"5040f0db-be61-4496-b978-27715767d629","Type":"ContainerDied","Data":"8f12908e2afa8a57ae2b9d5ea39e24c2940945a5ef7276952cdc7ccc02740ec4"}
Sep 19 08:25:07.675376 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.675358    2584 scope.go:117] "RemoveContainer" containerID="8f12908e2afa8a57ae2b9d5ea39e24c2940945a5ef7276952cdc7ccc02740ec4"
Sep 19 08:25:07.679875 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.679853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr" event={"ID":"7b3384b3-a1de-4edf-898e-dc3a77db41bf","Type":"ContainerStarted","Data":"f5f3bf529c11c4089a91874617bba6ad3862a1b50388baa87d7689fb13c424f9"}
Sep 19 08:25:07.692918 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.692874    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg" podStartSLOduration=1.81638228 podStartE2EDuration="4.692857826s" totalImagesPullingTime="2.876475546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:07.692121302 +0000 UTC m=+275.776622104" watchObservedRunningTime="2026-09-19 08:25:07.692857826 +0000 UTC m=+275.777358630"
Sep 19 08:25:07.725270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:07.725231    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr" podStartSLOduration=1.91229933 podStartE2EDuration="4.725216477s" totalImagesPullingTime="2.812917147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:07.724207288 +0000 UTC m=+275.808708094" watchObservedRunningTime="2026-09-19 08:25:07.725216477 +0000 UTC m=+275.809717327"
Sep 19 08:25:08.684418 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.684388    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/1.log"
Sep 19 08:25:08.684849 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.684831    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/0.log"
Sep 19 08:25:08.684894 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.684868    2584 generic.go:356] "Generic (PLEG): container finished" podID="5040f0db-be61-4496-b978-27715767d629" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea" exitCode=255
Sep 19 08:25:08.684932 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.684898    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" event={"ID":"5040f0db-be61-4496-b978-27715767d629","Type":"ContainerDied","Data":"023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"}
Sep 19 08:25:08.685005 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.684973    2584 scope.go:117] "RemoveContainer" containerID="8f12908e2afa8a57ae2b9d5ea39e24c2940945a5ef7276952cdc7ccc02740ec4"
Sep 19 08:25:08.685214 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.685185    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:08.685410 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:08.685370    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:08.777902 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.777874    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-drjbp"]
Sep 19 08:25:08.782188 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.782163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:08.784721 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.784698    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:25:08.784844 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.784704    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:25:08.784844 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.784742    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mq7r\""
Sep 19 08:25:08.792563 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.792541    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drjbp"]
Sep 19 08:25:08.941859 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.941781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60649c37-eb40-4171-8c6e-6de49dd4224c-crio-socket\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:08.941859 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.941824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:08.942088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.941935    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjfv\" (UniqueName: \"kubernetes.io/projected/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-api-access-wvjfv\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:08.942088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.942009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:08.942088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:08.942076    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60649c37-eb40-4171-8c6e-6de49dd4224c-data-volume\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042555 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60649c37-eb40-4171-8c6e-6de49dd4224c-data-volume\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042662 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60649c37-eb40-4171-8c6e-6de49dd4224c-crio-socket\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042662 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042777 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjfv\" (UniqueName: \"kubernetes.io/projected/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-api-access-wvjfv\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042777 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042777 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.042705    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.042777 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042756    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60649c37-eb40-4171-8c6e-6de49dd4224c-crio-socket\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.042777 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.042771    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:09.542751183 +0000 UTC m=+277.627251964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.042968 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.042802    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.042968 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.042856    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:09.542839458 +0000 UTC m=+277.627340256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.042968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.042870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60649c37-eb40-4171-8c6e-6de49dd4224c-data-volume\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.050322 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.050297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjfv\" (UniqueName: \"kubernetes.io/projected/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-api-access-wvjfv\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.547665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.547636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.547811 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.547711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:09.547811 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.547748    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.547811 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.547803    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.547921 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.547805    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:10.54778907 +0000 UTC m=+278.632289857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:09.547921 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.547852    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:10.547840363 +0000 UTC m=+278.632341144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:09.689313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.689289    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/1.log"
Sep 19 08:25:09.689653 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.689584    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:09.689731 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:09.689713    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:09.770644 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.770620    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vdn9"]
Sep 19 08:25:09.773618 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.773604    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:09.775855 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.775832    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 19 08:25:09.775995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.775861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kvc6v\""
Sep 19 08:25:09.775995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.775837    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 19 08:25:09.776145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.776128    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 19 08:25:09.776197 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.776155    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 19 08:25:09.781521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.781501    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vdn9"]
Sep 19 08:25:09.951511 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.951444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:09.951632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.951550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnfp\" (UniqueName: \"kubernetes.io/projected/ae178bc2-ef25-4aa1-bed2-46428ab4b744-kube-api-access-vgnfp\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:09.951692 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:09.951665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-key\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.052308 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.052281    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnfp\" (UniqueName: \"kubernetes.io/projected/ae178bc2-ef25-4aa1-bed2-46428ab4b744-kube-api-access-vgnfp\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.052412 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.052352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-key\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.052412 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.052395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.052968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.052950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.054495 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.054474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ae178bc2-ef25-4aa1-bed2-46428ab4b744-signing-key\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.060085 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.060067    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnfp\" (UniqueName: \"kubernetes.io/projected/ae178bc2-ef25-4aa1-bed2-46428ab4b744-kube-api-access-vgnfp\") pod \"service-ca-55f6fc7b86-4vdn9\" (UID: \"ae178bc2-ef25-4aa1-bed2-46428ab4b744\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.083122 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.083096    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9"
Sep 19 08:25:10.350751 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.350725    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vdn9"]
Sep 19 08:25:10.353262 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:10.353239    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae178bc2_ef25_4aa1_bed2_46428ab4b744.slice/crio-d2888be01f18fc54db200ad9cdf6357ca881a707376b02115e3b7e481c0e5d60 WatchSource:0}: Error finding container d2888be01f18fc54db200ad9cdf6357ca881a707376b02115e3b7e481c0e5d60: Status 404 returned error can't find the container with id d2888be01f18fc54db200ad9cdf6357ca881a707376b02115e3b7e481c0e5d60
Sep 19 08:25:10.555637 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.555608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:10.555740 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.555682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:10.555740 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:10.555730    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:10.555810 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:10.555777    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:10.555810 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:10.555786    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:12.555771627 +0000 UTC m=+280.640272413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:10.555890 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:10.555813    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:12.55580206 +0000 UTC m=+280.640302840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:10.558229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.558215    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2tqn_0be6872e-0257-4d2a-8943-eec60b49bd88/dns-node-resolver/0.log"
Sep 19 08:25:10.693048 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:10.692968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9" event={"ID":"ae178bc2-ef25-4aa1-bed2-46428ab4b744","Type":"ContainerStarted","Data":"d2888be01f18fc54db200ad9cdf6357ca881a707376b02115e3b7e481c0e5d60"}
Sep 19 08:25:11.262331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:11.262296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:11.262505 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:11.262350    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:11.262505 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:11.262413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:11.262505 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.262447    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:25:11.262505 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.262504    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:19.262487789 +0000 UTC m=+287.346988590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : secret "router-metrics-certs-default" not found
Sep 19 08:25:11.262696 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.262518    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:11.262696 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.262546    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:19.262531677 +0000 UTC m=+287.347032480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:11.262696 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.262572    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:19.26255733 +0000 UTC m=+287.347058117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:11.362803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:11.362769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:11.362977 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.362913    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:25:11.363074 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:11.362993    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls podName:3ffa901d-4b87-410b-96e3-e02b97be8fda nodeName:}" failed. No retries permitted until 2026-09-19 08:25:19.362957638 +0000 UTC m=+287.447458427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-z5jq2" (UID: "3ffa901d-4b87-410b-96e3-e02b97be8fda") : secret "samples-operator-tls" not found
Sep 19 08:25:11.757879 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:11.757851    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmzgn_028b379e-a3d7-4094-908c-e51d0ce4d1ff/node-ca/0.log"
Sep 19 08:25:12.571470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:12.571442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:12.571637 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:12.571517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:12.571637 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:12.571581    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:12.571721 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:12.571637    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:12.571721 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:12.571661    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:16.571639349 +0000 UTC m=+284.656140135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:12.571721 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:12.571678    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:16.571667721 +0000 UTC m=+284.656168522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:12.701842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:12.701816    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9" event={"ID":"ae178bc2-ef25-4aa1-bed2-46428ab4b744","Type":"ContainerStarted","Data":"83e88e67ceda9f3b823e31e5e84b45654f783b78160e11eb99f0626dc169f09c"}
Sep 19 08:25:12.717568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:12.717530    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-4vdn9" podStartSLOduration=2.125720751 podStartE2EDuration="3.717517862s" totalImagesPullingTime="1.591797111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:12.71639068 +0000 UTC m=+280.800891484" watchObservedRunningTime="2026-09-19 08:25:12.717517862 +0000 UTC m=+280.802018665"
Sep 19 08:25:12.758931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:12.758909    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmlcm_abb7958d-0540-407c-ba3e-f9d28f8115af/serve-healthcheck-canary/0.log"
Sep 19 08:25:13.386770 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:13.386731    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6778975687-jmfgj" podUID="93d30278-e75d-42cd-b133-63d05821ab87"
Sep 19 08:25:13.386770 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:13.386747    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" podUID="8e683a50-2d02-4b9b-b0cb-8744e2790ef9"
Sep 19 08:25:13.386959 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:13.386747    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vz85" podUID="6da27a61-af20-4407-934d-eda26bccb1b6"
Sep 19 08:25:13.704951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.704883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:25:13.704951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.704907    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:25:13.704951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.704889    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:25:13.775036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.775014    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:13.775430 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.775065    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:13.775430 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:13.775374    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:13.775555 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:13.775536    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:14.707675 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:14.707651    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:14.707849 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:14.707789    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:16.603592 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:16.603560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:16.604034 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:16.603641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:16.604034 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:16.603674    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:25:16.604034 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:16.603716    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:16.604034 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:16.603730    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:24.603715759 +0000 UTC m=+292.688216556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : configmap "kube-rbac-proxy" not found
Sep 19 08:25:16.604034 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:16.603755    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls podName:60649c37-eb40-4171-8c6e-6de49dd4224c nodeName:}" failed. No retries permitted until 2026-09-19 08:25:24.603744225 +0000 UTC m=+292.688245005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drjbp" (UID: "60649c37-eb40-4171-8c6e-6de49dd4224c") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:25:17.210001 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:17.209953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:25:17.210161 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:17.210044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") pod \"image-registry-6778975687-jmfgj\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") " pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:25:17.210161 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.210085    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:25:17.210161 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.210147    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert podName:8e683a50-2d02-4b9b-b0cb-8744e2790ef9 nodeName:}" failed. No retries permitted until 2026-09-19 08:27:19.210126781 +0000 UTC m=+407.294627578 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4tts9" (UID: "8e683a50-2d02-4b9b-b0cb-8744e2790ef9") : secret "networking-console-plugin-cert" not found
Sep 19 08:25:17.210290 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.210182    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:25:17.210290 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.210198    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6778975687-jmfgj: secret "image-registry-tls" not found
Sep 19 08:25:17.210290 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.210239    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls podName:93d30278-e75d-42cd-b133-63d05821ab87 nodeName:}" failed. No retries permitted until 2026-09-19 08:27:19.2102298 +0000 UTC m=+407.294730585 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls") pod "image-registry-6778975687-jmfgj" (UID: "93d30278-e75d-42cd-b133-63d05821ab87") : secret "image-registry-tls" not found
Sep 19 08:25:17.311145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:17.311113    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:25:17.311283 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.311231    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:25:17.311283 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:17.311276    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls podName:6da27a61-af20-4407-934d-eda26bccb1b6 nodeName:}" failed. No retries permitted until 2026-09-19 08:27:19.311263112 +0000 UTC m=+407.395763897 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls") pod "dns-default-4vz85" (UID: "6da27a61-af20-4407-934d-eda26bccb1b6") : secret "dns-default-metrics-tls" not found
Sep 19 08:25:19.325691 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.325658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:19.326128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.325700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:19.326128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.325752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:19.326128 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:19.325886    2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:19.326128 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:19.325947    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle podName:1211f7b0-61e1-4389-854d-0fef63b6c75f nodeName:}" failed. No retries permitted until 2026-09-19 08:25:35.32593105 +0000 UTC m=+303.410431848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle") pod "router-default-9b7b7b666-vbwq7" (UID: "1211f7b0-61e1-4389-854d-0fef63b6c75f") : configmap references non-existent config key: service-ca.crt
Sep 19 08:25:19.326128 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:19.325962    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls podName:cc1b5955-9ae6-42a9-8526-a14abb8a9c64 nodeName:}" failed. No retries permitted until 2026-09-19 08:25:35.325956228 +0000 UTC m=+303.410457009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-dvbd9" (UID: "cc1b5955-9ae6-42a9-8526-a14abb8a9c64") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:25:19.328191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.328173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1211f7b0-61e1-4389-854d-0fef63b6c75f-metrics-certs\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:19.426571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.426542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:19.428800 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.428782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ffa901d-4b87-410b-96e3-e02b97be8fda-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-z5jq2\" (UID: \"3ffa901d-4b87-410b-96e3-e02b97be8fda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:19.469655 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.469629    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"
Sep 19 08:25:19.744331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:19.744305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2"]
Sep 19 08:25:20.734280 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:20.734246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2" event={"ID":"3ffa901d-4b87-410b-96e3-e02b97be8fda","Type":"ContainerStarted","Data":"c569ca365a4e0d45bab499c0e7bf1defcffe65d6362ae04ef4312379c75d08ef"}
Sep 19 08:25:21.739207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:21.739178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2" event={"ID":"3ffa901d-4b87-410b-96e3-e02b97be8fda","Type":"ContainerStarted","Data":"703b9ab08cd5ad18d5289243f52920e0a2ef115168fbaf68b99535d389df8839"}
Sep 19 08:25:21.739465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:21.739236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2" event={"ID":"3ffa901d-4b87-410b-96e3-e02b97be8fda","Type":"ContainerStarted","Data":"e2cdbb9f0da86b973a58748fe131cfdd14bf6667a70ec447e92193279ff8ff6e"}
Sep 19 08:25:21.759715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:21.759673    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-z5jq2" podStartSLOduration=16.909707907 podStartE2EDuration="18.759657996s" totalImagesPullingTime="1.849950089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:21.757641199 +0000 UTC m=+289.842142001" watchObservedRunningTime="2026-09-19 08:25:21.759657996 +0000 UTC m=+289.844158799"
Sep 19 08:25:24.671087 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.671053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:24.671447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.671122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:24.671614 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.671595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60649c37-eb40-4171-8c6e-6de49dd4224c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:24.673454 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.673428    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60649c37-eb40-4171-8c6e-6de49dd4224c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drjbp\" (UID: \"60649c37-eb40-4171-8c6e-6de49dd4224c\") " pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:24.691346 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.691321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drjbp"
Sep 19 08:25:24.963243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:24.963210    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drjbp"]
Sep 19 08:25:24.967802 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:24.967774    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60649c37_eb40_4171_8c6e_6de49dd4224c.slice/crio-84478a7abecb98bf04442fed2061b6bd7f36f61aec9b5206de56800180bd0490 WatchSource:0}: Error finding container 84478a7abecb98bf04442fed2061b6bd7f36f61aec9b5206de56800180bd0490: Status 404 returned error can't find the container with id 84478a7abecb98bf04442fed2061b6bd7f36f61aec9b5206de56800180bd0490
Sep 19 08:25:25.754308 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:25.754285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drjbp" event={"ID":"60649c37-eb40-4171-8c6e-6de49dd4224c","Type":"ContainerStarted","Data":"22ede333bf8b92fb117951874ba7435294781d0e816e29c128a7c38b2c2fe168"}
Sep 19 08:25:25.754568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:25.754334    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drjbp" event={"ID":"60649c37-eb40-4171-8c6e-6de49dd4224c","Type":"ContainerStarted","Data":"2109f032772fc5c41a71aef76e35372b410f85d94f51543be9bda758b45f0b53"}
Sep 19 08:25:25.754568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:25.754357    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drjbp" event={"ID":"60649c37-eb40-4171-8c6e-6de49dd4224c","Type":"ContainerStarted","Data":"84478a7abecb98bf04442fed2061b6bd7f36f61aec9b5206de56800180bd0490"}
Sep 19 08:25:27.764376 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:27.764332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drjbp" event={"ID":"60649c37-eb40-4171-8c6e-6de49dd4224c","Type":"ContainerStarted","Data":"c9c2c035687b5693e1d615453471f5cb9be9c46a3d8f5cc07a1974ade0b7b89a"}
Sep 19 08:25:27.782149 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:27.782097    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-drjbp" podStartSLOduration=17.839906841 podStartE2EDuration="19.782082601s" totalImagesPullingTime="1.94217576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:25:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:27.78157778 +0000 UTC m=+295.866078600" watchObservedRunningTime="2026-09-19 08:25:27.782082601 +0000 UTC m=+295.866583404"
Sep 19 08:25:27.902259 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:27.902226    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-s4btv" podUID="d0b18af0-056d-42a2-aee2-e0fb3d43a166"
Sep 19 08:25:29.901712 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:29.901684    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:30.776887 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.776860    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:25:30.777303 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.777288    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/1.log"
Sep 19 08:25:30.777373 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.777318    2584 generic.go:356] "Generic (PLEG): container finished" podID="5040f0db-be61-4496-b978-27715767d629" containerID="9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f" exitCode=255
Sep 19 08:25:30.777407 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.777385    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" event={"ID":"5040f0db-be61-4496-b978-27715767d629","Type":"ContainerDied","Data":"9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f"}
Sep 19 08:25:30.777440 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.777431    2584 scope.go:117] "RemoveContainer" containerID="023425261d7f89f42c724c2734924a42f4811f5db4711660604bdc7082e83bea"
Sep 19 08:25:30.777758 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:30.777733    2584 scope.go:117] "RemoveContainer" containerID="9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f"
Sep 19 08:25:30.777925 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:30.777904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:31.782912 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:31.782878    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:25:33.774909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.774871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:33.775336 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.774968    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:25:33.775336 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.775312    2584 scope.go:117] "RemoveContainer" containerID="9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f"
Sep 19 08:25:33.775547 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:33.775522    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:33.853929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.853901    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:25:33.854542 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.854519    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:25:33.860042 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.860018    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:25:33.860574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.860557    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:25:33.864784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:33.864765    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:25:35.351638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.351599    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:35.356810 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.351663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:35.356810 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.352258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1211f7b0-61e1-4389-854d-0fef63b6c75f-service-ca-bundle\") pod \"router-default-9b7b7b666-vbwq7\" (UID: \"1211f7b0-61e1-4389-854d-0fef63b6c75f\") " pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:35.356810 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.354138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc1b5955-9ae6-42a9-8526-a14abb8a9c64-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-dvbd9\" (UID: \"cc1b5955-9ae6-42a9-8526-a14abb8a9c64\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:35.568523 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.568494    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fl5gq\""
Sep 19 08:25:35.576380 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.576362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"
Sep 19 08:25:35.585880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.585861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z9qkd\""
Sep 19 08:25:35.594660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.594640    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:35.843876 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.843843    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9"]
Sep 19 08:25:35.846498 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:35.846474    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1b5955_9ae6_42a9_8526_a14abb8a9c64.slice/crio-c5a80c26639e4ceefd66a61c4df64e4d27840a3ddd9518e89e14bc6d3cb30aac WatchSource:0}: Error finding container c5a80c26639e4ceefd66a61c4df64e4d27840a3ddd9518e89e14bc6d3cb30aac: Status 404 returned error can't find the container with id c5a80c26639e4ceefd66a61c4df64e4d27840a3ddd9518e89e14bc6d3cb30aac
Sep 19 08:25:35.848242 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.848227    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:25:35.869234 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:35.869166    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9b7b7b666-vbwq7"]
Sep 19 08:25:35.872380 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:35.872359    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1211f7b0_61e1_4389_854d_0fef63b6c75f.slice/crio-c9977d8aa15b344104773fd5ce6ee1e4a2d2beb38a431e580ca8da1cb1860667 WatchSource:0}: Error finding container c9977d8aa15b344104773fd5ce6ee1e4a2d2beb38a431e580ca8da1cb1860667: Status 404 returned error can't find the container with id c9977d8aa15b344104773fd5ce6ee1e4a2d2beb38a431e580ca8da1cb1860667
Sep 19 08:25:36.801877 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:36.801823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9b7b7b666-vbwq7" event={"ID":"1211f7b0-61e1-4389-854d-0fef63b6c75f","Type":"ContainerStarted","Data":"0094aee8953e405715c613205f171fa97e6d36f9569cd0abd7fb81c61e631645"}
Sep 19 08:25:36.802311 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:36.801891    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9b7b7b666-vbwq7" event={"ID":"1211f7b0-61e1-4389-854d-0fef63b6c75f","Type":"ContainerStarted","Data":"c9977d8aa15b344104773fd5ce6ee1e4a2d2beb38a431e580ca8da1cb1860667"}
Sep 19 08:25:36.803095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:36.803068    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9" event={"ID":"cc1b5955-9ae6-42a9-8526-a14abb8a9c64","Type":"ContainerStarted","Data":"c5a80c26639e4ceefd66a61c4df64e4d27840a3ddd9518e89e14bc6d3cb30aac"}
Sep 19 08:25:36.823615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:36.823567    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-9b7b7b666-vbwq7" podStartSLOduration=33.823543777 podStartE2EDuration="33.823543777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:36.823273418 +0000 UTC m=+304.907774235" watchObservedRunningTime="2026-09-19 08:25:36.823543777 +0000 UTC m=+304.908044579"
Sep 19 08:25:37.595085 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.595007    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:37.597372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.597351    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:37.807713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.807682    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9" event={"ID":"cc1b5955-9ae6-42a9-8526-a14abb8a9c64","Type":"ContainerStarted","Data":"40cc4db7aef9300c3772b09640dc39c57da41ccb73f81a8b982755e432aed5d4"}
Sep 19 08:25:37.808129 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.807852    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:37.809592 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.809548    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9b7b7b666-vbwq7"
Sep 19 08:25:37.828600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:37.828552    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-dvbd9" podStartSLOduration=33.332860234 podStartE2EDuration="34.82853515s" totalImagesPullingTime="1.495674916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:37.828482659 +0000 UTC m=+305.912983474" watchObservedRunningTime="2026-09-19 08:25:37.82853515 +0000 UTC m=+305.913035954"
Sep 19 08:25:41.904214 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:41.904180    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:25:46.192556 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.192516    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"]
Sep 19 08:25:46.194869 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.194843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.197359 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.197334    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 19 08:25:46.198037 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.198016    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 19 08:25:46.198153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.198023    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:25:46.198153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.198021    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8ck7f\""
Sep 19 08:25:46.207782 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.207760    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"]
Sep 19 08:25:46.220557 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.220536    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hfskq"]
Sep 19 08:25:46.223060 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.223044    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.224974 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.224957    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:25:46.225396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.225377    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:25:46.225595 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.225433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97wlz\""
Sep 19 08:25:46.225595 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.225472    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:25:46.331414 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331539 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0eb968-fb92-4a9f-acb1-260b42eddf27-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.331539 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.331539 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsqtw\" (UniqueName: \"kubernetes.io/projected/df0eb968-fb92-4a9f-acb1-260b42eddf27-kube-api-access-bsqtw\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.331705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-sys\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.331705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-root\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/734315ee-c009-4d73-8962-b3be234005f7-kube-api-access-wflqd\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-tls\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331800    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-metrics-client-ca\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-textfile\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.331929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.331867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-wtmp\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsqtw\" (UniqueName: \"kubernetes.io/projected/df0eb968-fb92-4a9f-acb1-260b42eddf27-kube-api-access-bsqtw\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433048    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-sys\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-root\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/734315ee-c009-4d73-8962-b3be234005f7-kube-api-access-wflqd\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-sys\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-tls\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-metrics-client-ca\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-textfile\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-wtmp\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0eb968-fb92-4a9f-acb1-260b42eddf27-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.433503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433834 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-textfile\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.433892 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.433831    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-root\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.434027 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.434006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.434213 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.434191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-wtmp\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.434656 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.434634    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0eb968-fb92-4a9f-acb1-260b42eddf27-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.435184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.435162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734315ee-c009-4d73-8962-b3be234005f7-metrics-client-ca\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.436053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.436032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.436685 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.436661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734315ee-c009-4d73-8962-b3be234005f7-node-exporter-tls\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.436822 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.436797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.437339 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.437322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/df0eb968-fb92-4a9f-acb1-260b42eddf27-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.442074 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.442050    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsqtw\" (UniqueName: \"kubernetes.io/projected/df0eb968-fb92-4a9f-acb1-260b42eddf27-kube-api-access-bsqtw\") pod \"openshift-state-metrics-86c5bc79d6-99f7r\" (UID: \"df0eb968-fb92-4a9f-acb1-260b42eddf27\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.445304 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.445247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflqd\" (UniqueName: \"kubernetes.io/projected/734315ee-c009-4d73-8962-b3be234005f7-kube-api-access-wflqd\") pod \"node-exporter-hfskq\" (UID: \"734315ee-c009-4d73-8962-b3be234005f7\") " pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.504852 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.504830    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"
Sep 19 08:25:46.533411 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.533390    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hfskq"
Sep 19 08:25:46.543607 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:46.543579    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734315ee_c009_4d73_8962_b3be234005f7.slice/crio-9341519b16da77b67295f8883cf38fc6abfa5650fc6a36d99e6e2cf5c836e7dc WatchSource:0}: Error finding container 9341519b16da77b67295f8883cf38fc6abfa5650fc6a36d99e6e2cf5c836e7dc: Status 404 returned error can't find the container with id 9341519b16da77b67295f8883cf38fc6abfa5650fc6a36d99e6e2cf5c836e7dc
Sep 19 08:25:46.775088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.775060    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r"]
Sep 19 08:25:46.777202 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:46.777178    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0eb968_fb92_4a9f_acb1_260b42eddf27.slice/crio-387e68f70ed72755a925885578131756ee3a763ecb8c6cfb5dc9a8192d1b7712 WatchSource:0}: Error finding container 387e68f70ed72755a925885578131756ee3a763ecb8c6cfb5dc9a8192d1b7712: Status 404 returned error can't find the container with id 387e68f70ed72755a925885578131756ee3a763ecb8c6cfb5dc9a8192d1b7712
Sep 19 08:25:46.838814 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.838790    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r" event={"ID":"df0eb968-fb92-4a9f-acb1-260b42eddf27","Type":"ContainerStarted","Data":"e287d15bdee9d6f5e4b0082f5034a4f797d7338258d0707044909906f57d7169"}
Sep 19 08:25:46.838910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.838852    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r" event={"ID":"df0eb968-fb92-4a9f-acb1-260b42eddf27","Type":"ContainerStarted","Data":"387e68f70ed72755a925885578131756ee3a763ecb8c6cfb5dc9a8192d1b7712"}
Sep 19 08:25:46.840185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.840161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfskq" event={"ID":"734315ee-c009-4d73-8962-b3be234005f7","Type":"ContainerStarted","Data":"9341519b16da77b67295f8883cf38fc6abfa5650fc6a36d99e6e2cf5c836e7dc"}
Sep 19 08:25:46.901950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:46.901881    2584 scope.go:117] "RemoveContainer" containerID="9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f"
Sep 19 08:25:46.902100 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:46.902075    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-6sw7w_openshift-console-operator(5040f0db-be61-4496-b978-27715767d629)\"" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podUID="5040f0db-be61-4496-b978-27715767d629"
Sep 19 08:25:47.290392 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.290361    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:25:47.293552 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.293530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.296784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.296741    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 08:25:47.296902 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.296803    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 08:25:47.296902 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.296883    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 08:25:47.297071 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.296750    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 08:25:47.297475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297207    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 08:25:47.297475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297210    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 08:25:47.297475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297238    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7kql2\""
Sep 19 08:25:47.297475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297255    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 08:25:47.297708 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297572    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 08:25:47.297768 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.297747    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 08:25:47.312071 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.312008    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:25:47.443017 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.442952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443297 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443297 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443297 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkw9q\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443679 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.443679 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.443560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545068 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545133    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkw9q\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545445    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.545711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.545551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.546565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.546192    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.546565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.546253    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.548603 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.548573    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.548703 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.548672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.549063 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.549039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.549671 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.549315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.549821 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.549608    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.549975 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.549954    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.550565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.550535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.550649 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.550542    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.551153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.551134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.556289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.556267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkw9q\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q\") pod \"alertmanager-main-0\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.605152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.605128    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:25:47.846560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.846478    2584 generic.go:356] "Generic (PLEG): container finished" podID="734315ee-c009-4d73-8962-b3be234005f7" containerID="d7538a89b8f62c1b0b5f1b36aedc5ae8a6b3e704fe2b6fb701d054eb0986e6c3" exitCode=0
Sep 19 08:25:47.846560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.846535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfskq" event={"ID":"734315ee-c009-4d73-8962-b3be234005f7","Type":"ContainerDied","Data":"d7538a89b8f62c1b0b5f1b36aedc5ae8a6b3e704fe2b6fb701d054eb0986e6c3"}
Sep 19 08:25:47.848793 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.848767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r" event={"ID":"df0eb968-fb92-4a9f-acb1-260b42eddf27","Type":"ContainerStarted","Data":"5635ce58287537a0b1afade00cd9cceeba6ab9c95b3c71c88fdc6425417c275c"}
Sep 19 08:25:47.889555 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:47.889506    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:25:48.004347 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:48.004315    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1e31fa_5433_40a5_86b0_706fb8e12b76.slice/crio-de1b92e5181413bd31e3ec0005a605e9f46e83ff2a449566b7a5697d12e01204 WatchSource:0}: Error finding container de1b92e5181413bd31e3ec0005a605e9f46e83ff2a449566b7a5697d12e01204: Status 404 returned error can't find the container with id de1b92e5181413bd31e3ec0005a605e9f46e83ff2a449566b7a5697d12e01204
Sep 19 08:25:48.856099 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.856062    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfskq" event={"ID":"734315ee-c009-4d73-8962-b3be234005f7","Type":"ContainerStarted","Data":"61e1521c9a9c2c862fb34e3ddc5d8277edf891abd3c04a3b06878b5a18519705"}
Sep 19 08:25:48.856528 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.856124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hfskq" event={"ID":"734315ee-c009-4d73-8962-b3be234005f7","Type":"ContainerStarted","Data":"3daae26492aa2eda9966781a3d475a82716a669a001d731d91351589c1fbe617"}
Sep 19 08:25:48.858550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.858514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r" event={"ID":"df0eb968-fb92-4a9f-acb1-260b42eddf27","Type":"ContainerStarted","Data":"5674bf02951d699877ba46def61a88465e5e677021c7eb525578fd77bf064f28"}
Sep 19 08:25:48.860091 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.860064    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"de1b92e5181413bd31e3ec0005a605e9f46e83ff2a449566b7a5697d12e01204"}
Sep 19 08:25:48.881500 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.881430    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hfskq" podStartSLOduration=1.9226522400000001 podStartE2EDuration="2.881405714s" totalImagesPullingTime="893.589499ms" totalInitContainerRuntime="65.163975ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:48.879350229 +0000 UTC m=+316.963851045" watchObservedRunningTime="2026-09-19 08:25:48.881405714 +0000 UTC m=+316.965906516"
Sep 19 08:25:48.904228 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:48.904173    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-99f7r" podStartSLOduration=1.7494471470000001 podStartE2EDuration="2.904153826s" totalImagesPullingTime="1.154706679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:48.903542103 +0000 UTC m=+316.988042907" watchObservedRunningTime="2026-09-19 08:25:48.904153826 +0000 UTC m=+316.988654630"
Sep 19 08:25:49.864944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:49.864911    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a" exitCode=0
Sep 19 08:25:49.865338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:49.864953    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"}
Sep 19 08:25:50.975035 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:50.975004    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"]
Sep 19 08:25:50.978191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:50.978164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:50.980438 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:50.980416    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 19 08:25:50.980569 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:50.980422    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p68rl\""
Sep 19 08:25:50.986266 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:50.986210    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"]
Sep 19 08:25:51.076743 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.076714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbc9\" (UniqueName: \"kubernetes.io/projected/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-kube-api-access-ccbc9\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.076898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.076799    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.177922 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.177889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbc9\" (UniqueName: \"kubernetes.io/projected/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-kube-api-access-ccbc9\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.178097 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.178012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.180498 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.180470    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.185600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.185575    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbc9\" (UniqueName: \"kubernetes.io/projected/66d50cfc-f9e2-4955-9d5d-1214b37b5ba1-kube-api-access-ccbc9\") pod \"monitoring-plugin-5984459ff8-prfk7\" (UID: \"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.288599 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.288577    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:51.561206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.561179    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"]
Sep 19 08:25:51.563939 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:25:51.563914    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d50cfc_f9e2_4955_9d5d_1214b37b5ba1.slice/crio-9175ee396235c3ca54bb94a11f8e5befc86daf776d9a3fe679bd4f4e3a9148b1 WatchSource:0}: Error finding container 9175ee396235c3ca54bb94a11f8e5befc86daf776d9a3fe679bd4f4e3a9148b1: Status 404 returned error can't find the container with id 9175ee396235c3ca54bb94a11f8e5befc86daf776d9a3fe679bd4f4e3a9148b1
Sep 19 08:25:51.877532 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.877458    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"}
Sep 19 08:25:51.877737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.877546    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"}
Sep 19 08:25:51.877737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.877593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"}
Sep 19 08:25:51.877737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.877631    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"}
Sep 19 08:25:51.877737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.877672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"}
Sep 19 08:25:51.878802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:51.878767    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7" event={"ID":"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1","Type":"ContainerStarted","Data":"9175ee396235c3ca54bb94a11f8e5befc86daf776d9a3fe679bd4f4e3a9148b1"}
Sep 19 08:25:52.885404 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.885371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerStarted","Data":"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"}
Sep 19 08:25:52.886898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.886877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7" event={"ID":"66d50cfc-f9e2-4955-9d5d-1214b37b5ba1","Type":"ContainerStarted","Data":"c79e963d29b0a78357727a87508dd76dcb91a88ed8969c348ca664dd8e5adb76"}
Sep 19 08:25:52.887114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.887097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:52.891372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.891353    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7"
Sep 19 08:25:52.913708 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.913665    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.887464186 podStartE2EDuration="5.913652769s" totalImagesPullingTime="3.007926531s" totalInitContainerRuntime="18.262052ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:52.912206947 +0000 UTC m=+320.996707750" watchObservedRunningTime="2026-09-19 08:25:52.913652769 +0000 UTC m=+320.998153572"
Sep 19 08:25:52.931067 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:52.931030    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-prfk7" podStartSLOduration=1.7424120159999998 podStartE2EDuration="2.931016804s" totalImagesPullingTime="1.188604788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:52.930015609 +0000 UTC m=+321.014516409" watchObservedRunningTime="2026-09-19 08:25:52.931016804 +0000 UTC m=+321.015517604"
Sep 19 08:25:53.196165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:53.196099    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6778975687-jmfgj"]
Sep 19 08:25:53.196312 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:25:53.196294    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6778975687-jmfgj" podUID="93d30278-e75d-42cd-b133-63d05821ab87"
Sep 19 08:25:53.890759 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:53.890731    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:25:53.896700 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:53.896679    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:25:54.003473 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003452    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003594 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003489    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003594 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003594 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003531    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003594 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003549    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003610    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trvv9\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003853 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca\") pod \"93d30278-e75d-42cd-b133-63d05821ab87\" (UID: \"93d30278-e75d-42cd-b133-63d05821ab87\") "
Sep 19 08:25:54.003853 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.003823    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:25:54.005082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.004232    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:54.005082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.004869    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93d30278-e75d-42cd-b133-63d05821ab87-ca-trust-extracted\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.005256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.005084    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:54.006222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.006202    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:54.006326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.006229    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:54.006326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.006298    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9" (OuterVolumeSpecName: "kube-api-access-trvv9") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "kube-api-access-trvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:54.006326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.006306    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "93d30278-e75d-42cd-b133-63d05821ab87" (UID: "93d30278-e75d-42cd-b133-63d05821ab87"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:54.105852 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105832    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-trusted-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.105852 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105852    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93d30278-e75d-42cd-b133-63d05821ab87-registry-certificates\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.105977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105861    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-image-registry-private-configuration\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.105977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105870    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93d30278-e75d-42cd-b133-63d05821ab87-installation-pull-secrets\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.105977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105878    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-bound-sa-token\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.105977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.105887    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvv9\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-kube-api-access-trvv9\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:54.894108 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.894077    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6778975687-jmfgj"
Sep 19 08:25:54.928076 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.928050    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6778975687-jmfgj"]
Sep 19 08:25:54.932172 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:54.932152    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6778975687-jmfgj"]
Sep 19 08:25:55.114219 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:55.114193    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93d30278-e75d-42cd-b133-63d05821ab87-registry-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:25:55.906490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:55.906454    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93d30278-e75d-42cd-b133-63d05821ab87" path="/var/lib/kubelet/pods/93d30278-e75d-42cd-b133-63d05821ab87/volumes"
Sep 19 08:25:59.901724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:25:59.901688    2584 scope.go:117] "RemoveContainer" containerID="9f9a33d32f15a74d9cf71473ff5267bf46963391824a522452bd8e6a4707d77f"
Sep 19 08:26:00.812064 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.812029    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-45z8n"]
Sep 19 08:26:00.814219 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.814196    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:00.816517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.816497    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 19 08:26:00.816647 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.816499    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kb9gq\""
Sep 19 08:26:00.816647 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.816536    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 19 08:26:00.823628 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.823608    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-45z8n"]
Sep 19 08:26:00.918994 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.918955    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:26:00.919416 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.919037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" event={"ID":"5040f0db-be61-4496-b978-27715767d629","Type":"ContainerStarted","Data":"188e8d33727e45e51f73769a52b1820b3a318f98b3fdf7893db43caefe725441"}
Sep 19 08:26:00.919416 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.919293    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:26:00.924049 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.924025    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w"
Sep 19 08:26:00.937198 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.937139    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-6sw7w" podStartSLOduration=55.032698008 podStartE2EDuration="57.937120859s" totalImagesPullingTime="2.904422851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:00.935677854 +0000 UTC m=+329.020178657" watchObservedRunningTime="2026-09-19 08:26:00.937120859 +0000 UTC m=+329.021621736"
Sep 19 08:26:00.960996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:00.960956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7grj\" (UniqueName: \"kubernetes.io/projected/396b430b-43ea-45c6-9d2a-4c1d83813093-kube-api-access-q7grj\") pod \"downloads-6594f4c75-45z8n\" (UID: \"396b430b-43ea-45c6-9d2a-4c1d83813093\") " pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:01.061869 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:01.061843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7grj\" (UniqueName: \"kubernetes.io/projected/396b430b-43ea-45c6-9d2a-4c1d83813093-kube-api-access-q7grj\") pod \"downloads-6594f4c75-45z8n\" (UID: \"396b430b-43ea-45c6-9d2a-4c1d83813093\") " pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:01.070704 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:01.070645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7grj\" (UniqueName: \"kubernetes.io/projected/396b430b-43ea-45c6-9d2a-4c1d83813093-kube-api-access-q7grj\") pod \"downloads-6594f4c75-45z8n\" (UID: \"396b430b-43ea-45c6-9d2a-4c1d83813093\") " pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:01.124376 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:01.124352    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:01.396480 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:01.396463    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-45z8n"]
Sep 19 08:26:01.398576 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:26:01.398540    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396b430b_43ea_45c6_9d2a_4c1d83813093.slice/crio-d4e507cd87666f5704c89d41900b3c816981899abe68d381247de6892390f875 WatchSource:0}: Error finding container d4e507cd87666f5704c89d41900b3c816981899abe68d381247de6892390f875: Status 404 returned error can't find the container with id d4e507cd87666f5704c89d41900b3c816981899abe68d381247de6892390f875
Sep 19 08:26:01.923279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:01.923241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-45z8n" event={"ID":"396b430b-43ea-45c6-9d2a-4c1d83813093","Type":"ContainerStarted","Data":"d4e507cd87666f5704c89d41900b3c816981899abe68d381247de6892390f875"}
Sep 19 08:26:10.472265 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.472231    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:10.475293 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.475273    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.477808 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.477783    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f62mb\""
Sep 19 08:26:10.479083 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.479056    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 19 08:26:10.479186 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.479167    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 19 08:26:10.479186 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.479174    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 19 08:26:10.479296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.479213    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 19 08:26:10.479296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.479182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 19 08:26:10.488063 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.488033    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:10.547187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.547295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.547295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.547432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.547494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.547549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.547494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zb2\" (UniqueName: \"kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.648670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.648804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.648804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zb2\" (UniqueName: \"kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.648804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648739    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.648804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.649021 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.648808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.649639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.649569    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.649639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.649586    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.649807 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.649749    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.651450 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.651422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.651640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.651624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.656893 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.656870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zb2\" (UniqueName: \"kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2\") pod \"console-8b45d75d-9xtfd\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") " pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:10.786720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:10.786654    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:17.431804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.431775    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:17.993771 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.993729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-45z8n" event={"ID":"396b430b-43ea-45c6-9d2a-4c1d83813093","Type":"ContainerStarted","Data":"835f2c70050924cbc5049a50ad1aeeba8e942e4a11bb09db681049d184f94597"}
Sep 19 08:26:17.994783 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.994756    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:17.997019 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.996976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b45d75d-9xtfd" event={"ID":"3b596092-3466-4ff0-b941-cdd08bbe5747","Type":"ContainerStarted","Data":"533ec67608c9554f8e387fba3aa309b00b53b7e2e7b2fb2b02f66f2a559fbe1d"}
Sep 19 08:26:17.999409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.999380    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b3384b3-a1de-4edf-898e-dc3a77db41bf" containerID="f5f3bf529c11c4089a91874617bba6ad3862a1b50388baa87d7689fb13c424f9" exitCode=0
Sep 19 08:26:17.999500 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.999450    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr" event={"ID":"7b3384b3-a1de-4edf-898e-dc3a77db41bf","Type":"ContainerDied","Data":"f5f3bf529c11c4089a91874617bba6ad3862a1b50388baa87d7689fb13c424f9"}
Sep 19 08:26:18.000149 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:17.999871    2584 scope.go:117] "RemoveContainer" containerID="f5f3bf529c11c4089a91874617bba6ad3862a1b50388baa87d7689fb13c424f9"
Sep 19 08:26:18.007520 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.007457    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-45z8n"
Sep 19 08:26:18.034028 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.033699    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-45z8n" podStartSLOduration=2.184379886 podStartE2EDuration="18.033680389s" totalImagesPullingTime="15.849300503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:18.01433321 +0000 UTC m=+346.098834016" watchObservedRunningTime="2026-09-19 08:26:18.033680389 +0000 UTC m=+346.118181194"
Sep 19 08:26:18.532056 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.530585    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:26:18.536396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.536374    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.548476 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.548417    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:26:18.557945 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.556342    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629394    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629475    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.629803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.629662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkk2c\" (UniqueName: \"kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkk2c\" (UniqueName: \"kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730792    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.730958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.731832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.733559 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.733209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.735705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.734852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.735705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.735650    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.738941 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.738897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.758940 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.758889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkk2c\" (UniqueName: \"kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.759494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.759452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert\") pod \"console-6b54fd5f79-p59wt\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") " pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:18.857475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:18.857161    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:19.032761 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:19.032167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jx5qr" event={"ID":"7b3384b3-a1de-4edf-898e-dc3a77db41bf","Type":"ContainerStarted","Data":"4cd407299dda44a9e337d4b5c6d979c16a6a582ec7792648b5e25fe275848ab3"}
Sep 19 08:26:19.176121 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:19.176093    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:26:20.041286 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:20.041229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b54fd5f79-p59wt" event={"ID":"954752bb-b362-465e-a8e5-9a6547ed087f","Type":"ContainerStarted","Data":"c560d353404910cff15dc0f0e690bec5c0b155171fc619345649c58e1477a37b"}
Sep 19 08:26:24.064545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.064509    2584 generic.go:356] "Generic (PLEG): container finished" podID="97a94cfc-296d-4617-9970-fc8d7620c8b0" containerID="f1138d02c80cab2b5872ccf617b20798965f74dd09cdfc35ceb6d4ce4fc507fb" exitCode=0
Sep 19 08:26:24.065047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.064587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg" event={"ID":"97a94cfc-296d-4617-9970-fc8d7620c8b0","Type":"ContainerDied","Data":"f1138d02c80cab2b5872ccf617b20798965f74dd09cdfc35ceb6d4ce4fc507fb"}
Sep 19 08:26:24.065047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.065030    2584 scope.go:117] "RemoveContainer" containerID="f1138d02c80cab2b5872ccf617b20798965f74dd09cdfc35ceb6d4ce4fc507fb"
Sep 19 08:26:24.066598 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.066570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b54fd5f79-p59wt" event={"ID":"954752bb-b362-465e-a8e5-9a6547ed087f","Type":"ContainerStarted","Data":"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"}
Sep 19 08:26:24.068503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.068475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b45d75d-9xtfd" event={"ID":"3b596092-3466-4ff0-b941-cdd08bbe5747","Type":"ContainerStarted","Data":"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"}
Sep 19 08:26:24.100865 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.100786    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b45d75d-9xtfd" podStartSLOduration=7.893313492 podStartE2EDuration="14.100765364s" totalImagesPullingTime="6.207451872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:24.09946071 +0000 UTC m=+352.183961516" watchObservedRunningTime="2026-09-19 08:26:24.100765364 +0000 UTC m=+352.185266169"
Sep 19 08:26:24.130317 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:24.130272    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b54fd5f79-p59wt" podStartSLOduration=1.490469037 podStartE2EDuration="6.130253255s" totalImagesPullingTime="4.639784218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:24.128455832 +0000 UTC m=+352.212956627" watchObservedRunningTime="2026-09-19 08:26:24.130253255 +0000 UTC m=+352.214754057"
Sep 19 08:26:25.075784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:25.075751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mxmrg" event={"ID":"97a94cfc-296d-4617-9970-fc8d7620c8b0","Type":"ContainerStarted","Data":"4dc3a8eb42a850be7cafba014735900108006d97b1cd7df62b31833bf10ccabe"}
Sep 19 08:26:28.858550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:28.858504    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:28.859013 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:28.858564    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:28.863463 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:28.863441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:29.093950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:29.093924    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:26:29.140850 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:29.140789    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:30.787345 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:30.787309    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:47.871534 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:47.871488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:26:47.874054 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:47.874026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0b18af0-056d-42a2-aee2-e0fb3d43a166-metrics-certs\") pod \"network-metrics-daemon-s4btv\" (UID: \"d0b18af0-056d-42a2-aee2-e0fb3d43a166\") " pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:26:47.907818 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:47.907795    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-95gj6\""
Sep 19 08:26:47.916042 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:47.916024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s4btv"
Sep 19 08:26:48.206120 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:48.206045    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s4btv"]
Sep 19 08:26:48.209692 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:26:48.209661    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b18af0_056d_42a2_aee2_e0fb3d43a166.slice/crio-251281b3c74509d2eee4921d060b7469eb20d47d27382c8cc9e62c67f7500d22 WatchSource:0}: Error finding container 251281b3c74509d2eee4921d060b7469eb20d47d27382c8cc9e62c67f7500d22: Status 404 returned error can't find the container with id 251281b3c74509d2eee4921d060b7469eb20d47d27382c8cc9e62c67f7500d22
Sep 19 08:26:49.170119 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:49.170074    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4btv" event={"ID":"d0b18af0-056d-42a2-aee2-e0fb3d43a166","Type":"ContainerStarted","Data":"251281b3c74509d2eee4921d060b7469eb20d47d27382c8cc9e62c67f7500d22"}
Sep 19 08:26:50.176333 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:50.176295    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4btv" event={"ID":"d0b18af0-056d-42a2-aee2-e0fb3d43a166","Type":"ContainerStarted","Data":"e8d3af8a38c06c044ad8712c4a43ad5fde6b3e6599f30a45b935907943017607"}
Sep 19 08:26:50.176698 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:50.176355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s4btv" event={"ID":"d0b18af0-056d-42a2-aee2-e0fb3d43a166","Type":"ContainerStarted","Data":"2ddc934296a1de234af5ef9071cd7c2a1a9e505b6b3a7fa1175568da3dbc7124"}
Sep 19 08:26:50.198170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:50.198126    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s4btv" podStartSLOduration=374.695698177 podStartE2EDuration="6m16.198111394s" totalImagesPullingTime="1.502413217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:50.197570715 +0000 UTC m=+378.282071521" watchObservedRunningTime="2026-09-19 08:26:50.198111394 +0000 UTC m=+378.282612197"
Sep 19 08:26:54.161489 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.161448    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8b45d75d-9xtfd" podUID="3b596092-3466-4ff0-b941-cdd08bbe5747" containerName="console" containerID="cri-o://0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b" gracePeriod=15
Sep 19 08:26:54.435063 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.435003    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:54.524495 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524529    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524578    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524600    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9zb2\" (UniqueName: \"kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524851 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524753    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert\") pod \"3b596092-3466-4ff0-b941-cdd08bbe5747\" (UID: \"3b596092-3466-4ff0-b941-cdd08bbe5747\") "
Sep 19 08:26:54.524967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.524941    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config" (OuterVolumeSpecName: "console-config") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:26:54.525078 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.525033    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-console-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:54.525131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.525075    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:26:54.525257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.525240    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:26:54.526785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.526766    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:26:54.527176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.527159    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:26:54.527261 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.527171    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2" (OuterVolumeSpecName: "kube-api-access-l9zb2") pod "3b596092-3466-4ff0-b941-cdd08bbe5747" (UID: "3b596092-3466-4ff0-b941-cdd08bbe5747"). InnerVolumeSpecName "kube-api-access-l9zb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:26:54.625859 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.625835    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-oauth-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:54.625859 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.625855    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b596092-3466-4ff0-b941-cdd08bbe5747-console-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:54.625976 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.625865    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:54.625976 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.625875    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zb2\" (UniqueName: \"kubernetes.io/projected/3b596092-3466-4ff0-b941-cdd08bbe5747-kube-api-access-l9zb2\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:54.625976 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:54.625883    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b596092-3466-4ff0-b941-cdd08bbe5747-oauth-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:26:55.198716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.198681    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b596092-3466-4ff0-b941-cdd08bbe5747" containerID="0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b" exitCode=0
Sep 19 08:26:55.199132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.198736    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b45d75d-9xtfd"
Sep 19 08:26:55.199132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.198773    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b45d75d-9xtfd" event={"ID":"3b596092-3466-4ff0-b941-cdd08bbe5747","Type":"ContainerDied","Data":"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"}
Sep 19 08:26:55.199132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.198832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b45d75d-9xtfd" event={"ID":"3b596092-3466-4ff0-b941-cdd08bbe5747","Type":"ContainerDied","Data":"533ec67608c9554f8e387fba3aa309b00b53b7e2e7b2fb2b02f66f2a559fbe1d"}
Sep 19 08:26:55.199132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.198854    2584 scope.go:117] "RemoveContainer" containerID="0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"
Sep 19 08:26:55.207804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.207785    2584 scope.go:117] "RemoveContainer" containerID="0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"
Sep 19 08:26:55.208151 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:26:55.208127    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b\": container with ID starting with 0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b not found: ID does not exist" containerID="0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"
Sep 19 08:26:55.208235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.208158    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b"} err="failed to get container status \"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b\": rpc error: code = NotFound desc = could not find container \"0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b\": container with ID starting with 0a04779baef7552d4a06fec0a52e04b48cf0dee7dea9a661f3d3c948f288ec6b not found: ID does not exist"
Sep 19 08:26:55.220399 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.220371    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:55.222992 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.222955    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b45d75d-9xtfd"]
Sep 19 08:26:55.906153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:26:55.906123    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b596092-3466-4ff0-b941-cdd08bbe5747" path="/var/lib/kubelet/pods/3b596092-3466-4ff0-b941-cdd08bbe5747/volumes"
Sep 19 08:27:06.587472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587431    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:06.587914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587806    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="alertmanager" containerID="cri-o://132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2" gracePeriod=120
Sep 19 08:27:06.587914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587851    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-metric" containerID="cri-o://f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa" gracePeriod=120
Sep 19 08:27:06.587914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587867    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="config-reloader" containerID="cri-o://98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c" gracePeriod=120
Sep 19 08:27:06.588126 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587890    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-web" containerID="cri-o://bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35" gracePeriod=120
Sep 19 08:27:06.588126 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587939    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="prom-label-proxy" containerID="cri-o://80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31" gracePeriod=120
Sep 19 08:27:06.588126 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:06.587897    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy" containerID="cri-o://389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca" gracePeriod=120
Sep 19 08:27:07.251362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251327    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31" exitCode=0
Sep 19 08:27:07.251362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251358    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca" exitCode=0
Sep 19 08:27:07.251362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251367    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c" exitCode=0
Sep 19 08:27:07.251362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251373    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2" exitCode=0
Sep 19 08:27:07.251686 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251410    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"}
Sep 19 08:27:07.251686 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"}
Sep 19 08:27:07.251686 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"}
Sep 19 08:27:07.251686 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.251601    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"}
Sep 19 08:27:07.824669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.824645    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:07.926014 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.925930    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926014 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.925970    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926035    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926056    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926077    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkw9q\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926099    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926126    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926160    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926222 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926194    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926254    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926311    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926345    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy\") pod \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\" (UID: \"ee1e31fa-5433-40a5-86b0-706fb8e12b76\") "
Sep 19 08:27:07.926635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926436    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:07.926843 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926663    2584 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-metrics-client-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:07.926843 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.926771    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:07.927321 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.927280    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:07.929485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.929334    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:07.929627 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.929588    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out" (OuterVolumeSpecName: "config-out") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:07.929757 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.929630    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.929757 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.929733    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q" (OuterVolumeSpecName: "kube-api-access-qkw9q") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "kube-api-access-qkw9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:07.930026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.930007    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.930402 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.930381    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume" (OuterVolumeSpecName: "config-volume") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.930468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.930438    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.930931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.930913    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.933479 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.933444    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:07.939726 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:07.939705    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config" (OuterVolumeSpecName: "web-config") pod "ee1e31fa-5433-40a5-86b0-706fb8e12b76" (UID: "ee1e31fa-5433-40a5-86b0-706fb8e12b76"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:08.027120 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027100    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027120 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027119    2584 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-tls-assets\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027128    2584 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-web-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027136    2584 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-cluster-tls-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027146    2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-main-db\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027154    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkw9q\" (UniqueName: \"kubernetes.io/projected/ee1e31fa-5433-40a5-86b0-706fb8e12b76-kube-api-access-qkw9q\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027162    2584 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee1e31fa-5433-40a5-86b0-706fb8e12b76-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027170    2584 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-out\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027178    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027186    2584 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-config-volume\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027194    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.027229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.027202    2584 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee1e31fa-5433-40a5-86b0-706fb8e12b76-secret-alertmanager-main-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:08.258055 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258025    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa" exitCode=0
Sep 19 08:27:08.258055 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258050    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerID="bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35" exitCode=0
Sep 19 08:27:08.258171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258117    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.258206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"}
Sep 19 08:27:08.258264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"}
Sep 19 08:27:08.258298 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258282    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee1e31fa-5433-40a5-86b0-706fb8e12b76","Type":"ContainerDied","Data":"de1b92e5181413bd31e3ec0005a605e9f46e83ff2a449566b7a5697d12e01204"}
Sep 19 08:27:08.258333 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.258314    2584 scope.go:117] "RemoveContainer" containerID="80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"
Sep 19 08:27:08.265944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.265927    2584 scope.go:117] "RemoveContainer" containerID="f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"
Sep 19 08:27:08.274587 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.274570    2584 scope.go:117] "RemoveContainer" containerID="389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"
Sep 19 08:27:08.281167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.281144    2584 scope.go:117] "RemoveContainer" containerID="bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"
Sep 19 08:27:08.290909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.288456    2584 scope.go:117] "RemoveContainer" containerID="98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"
Sep 19 08:27:08.290909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.289558    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:08.294969 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.294923    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:08.300631 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.300615    2584 scope.go:117] "RemoveContainer" containerID="132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"
Sep 19 08:27:08.307289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.307270    2584 scope.go:117] "RemoveContainer" containerID="0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"
Sep 19 08:27:08.313621 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.313605    2584 scope.go:117] "RemoveContainer" containerID="80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"
Sep 19 08:27:08.313869 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.313850    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31\": container with ID starting with 80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31 not found: ID does not exist" containerID="80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"
Sep 19 08:27:08.313918 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.313877    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"} err="failed to get container status \"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31\": rpc error: code = NotFound desc = could not find container \"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31\": container with ID starting with 80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31 not found: ID does not exist"
Sep 19 08:27:08.313918 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.313894    2584 scope.go:117] "RemoveContainer" containerID="f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"
Sep 19 08:27:08.314180 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.314161    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa\": container with ID starting with f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa not found: ID does not exist" containerID="f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"
Sep 19 08:27:08.314231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314185    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"} err="failed to get container status \"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa\": rpc error: code = NotFound desc = could not find container \"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa\": container with ID starting with f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa not found: ID does not exist"
Sep 19 08:27:08.314231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314200    2584 scope.go:117] "RemoveContainer" containerID="389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"
Sep 19 08:27:08.314433 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.314415    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca\": container with ID starting with 389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca not found: ID does not exist" containerID="389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"
Sep 19 08:27:08.314478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314439    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"} err="failed to get container status \"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca\": rpc error: code = NotFound desc = could not find container \"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca\": container with ID starting with 389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca not found: ID does not exist"
Sep 19 08:27:08.314478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314452    2584 scope.go:117] "RemoveContainer" containerID="bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"
Sep 19 08:27:08.314679 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.314661    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35\": container with ID starting with bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35 not found: ID does not exist" containerID="bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"
Sep 19 08:27:08.314724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314685    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"} err="failed to get container status \"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35\": rpc error: code = NotFound desc = could not find container \"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35\": container with ID starting with bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35 not found: ID does not exist"
Sep 19 08:27:08.314724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314698    2584 scope.go:117] "RemoveContainer" containerID="98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"
Sep 19 08:27:08.314934 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.314916    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c\": container with ID starting with 98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c not found: ID does not exist" containerID="98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"
Sep 19 08:27:08.315058 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314940    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"} err="failed to get container status \"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c\": rpc error: code = NotFound desc = could not find container \"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c\": container with ID starting with 98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c not found: ID does not exist"
Sep 19 08:27:08.315058 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.314954    2584 scope.go:117] "RemoveContainer" containerID="132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"
Sep 19 08:27:08.315173 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.315158    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2\": container with ID starting with 132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2 not found: ID does not exist" containerID="132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"
Sep 19 08:27:08.315235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315175    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"} err="failed to get container status \"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2\": rpc error: code = NotFound desc = could not find container \"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2\": container with ID starting with 132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2 not found: ID does not exist"
Sep 19 08:27:08.315235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315187    2584 scope.go:117] "RemoveContainer" containerID="0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"
Sep 19 08:27:08.315406 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:08.315390    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a\": container with ID starting with 0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a not found: ID does not exist" containerID="0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"
Sep 19 08:27:08.315450 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315409    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"} err="failed to get container status \"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a\": rpc error: code = NotFound desc = could not find container \"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a\": container with ID starting with 0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a not found: ID does not exist"
Sep 19 08:27:08.315450 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315420    2584 scope.go:117] "RemoveContainer" containerID="80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"
Sep 19 08:27:08.315638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315621    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31"} err="failed to get container status \"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31\": rpc error: code = NotFound desc = could not find container \"80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31\": container with ID starting with 80ae455ac61d6e1bdbb9880c0bf34b3c7367db0240ad5219bbf8c148c2b7cc31 not found: ID does not exist"
Sep 19 08:27:08.315683 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315638    2584 scope.go:117] "RemoveContainer" containerID="f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"
Sep 19 08:27:08.315861 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315842    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa"} err="failed to get container status \"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa\": rpc error: code = NotFound desc = could not find container \"f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa\": container with ID starting with f7b2756bb0286008de9433ad6f6460f2de379970fda907d6d779aca5a1fd55aa not found: ID does not exist"
Sep 19 08:27:08.315903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.315862    2584 scope.go:117] "RemoveContainer" containerID="389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"
Sep 19 08:27:08.316118 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316100    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca"} err="failed to get container status \"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca\": rpc error: code = NotFound desc = could not find container \"389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca\": container with ID starting with 389c9d4fe52d66f59a13f6f27d50ca1abc3ffaffbf1a57c3d88ebe65d137a8ca not found: ID does not exist"
Sep 19 08:27:08.316118 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316117    2584 scope.go:117] "RemoveContainer" containerID="bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"
Sep 19 08:27:08.316331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316304    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35"} err="failed to get container status \"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35\": rpc error: code = NotFound desc = could not find container \"bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35\": container with ID starting with bd3c5afa9bfbfd6f139c2d77e626eedbfbaed365793bf07b66c6f46cb8006b35 not found: ID does not exist"
Sep 19 08:27:08.316331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316326    2584 scope.go:117] "RemoveContainer" containerID="98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"
Sep 19 08:27:08.316676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316654    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c"} err="failed to get container status \"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c\": rpc error: code = NotFound desc = could not find container \"98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c\": container with ID starting with 98473457fec5695c8e4ee974e7a770e3e3bced72e4fa4fbda4010308bd50d81c not found: ID does not exist"
Sep 19 08:27:08.316676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316674    2584 scope.go:117] "RemoveContainer" containerID="132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"
Sep 19 08:27:08.316910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316892    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2"} err="failed to get container status \"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2\": rpc error: code = NotFound desc = could not find container \"132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2\": container with ID starting with 132f8a4c24702be4457e003a7ee6438b88eb0136209d9b6a2e441d8233789df2 not found: ID does not exist"
Sep 19 08:27:08.316910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.316909    2584 scope.go:117] "RemoveContainer" containerID="0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"
Sep 19 08:27:08.317156 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.317137    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a"} err="failed to get container status \"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a\": rpc error: code = NotFound desc = could not find container \"0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a\": container with ID starting with 0200e17517f51018fe54b8dc101557edb41955f911f5a4de6d3ebd32f3f1b46a not found: ID does not exist"
Sep 19 08:27:08.329669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:08.329928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329917    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-web"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329930    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-web"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329937    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="prom-label-proxy"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329944    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="prom-label-proxy"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329953    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329958    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329966    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="alertmanager"
Sep 19 08:27:08.329973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329971    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="alertmanager"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329977    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="config-reloader"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.329997    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="config-reloader"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330002    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b596092-3466-4ff0-b941-cdd08bbe5747" containerName="console"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330007    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b596092-3466-4ff0-b941-cdd08bbe5747" containerName="console"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330018    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-metric"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330023    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-metric"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330031    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="init-config-reloader"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330036    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="init-config-reloader"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330079    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="config-reloader"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330087    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="alertmanager"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330092    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-web"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330099    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330103    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b596092-3466-4ff0-b941-cdd08bbe5747" containerName="console"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330108    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="prom-label-proxy"
Sep 19 08:27:08.330187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.330114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" containerName="kube-rbac-proxy-metric"
Sep 19 08:27:08.333597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.333584    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.336179 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336148    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 08:27:08.336300 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336215    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 08:27:08.336300 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336257    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 08:27:08.336300 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336263    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 08:27:08.336830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336811    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 08:27:08.336915 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.336845    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 08:27:08.337074 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.337058    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 08:27:08.337164 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.337153    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 08:27:08.337255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.337242    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7kql2\""
Sep 19 08:27:08.343510 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.343492    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 08:27:08.352343 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.352324    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:08.430573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65jh\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-kube-api-access-x65jh\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-config-volume\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430723 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430723 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-web-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430790 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430790 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430790 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430816    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-config-out\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430868    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.430963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.430943    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531264    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65jh\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-kube-api-access-x65jh\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-config-volume\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-web-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531370    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531406    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531470 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-config-out\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.531802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.531530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.532030 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.532008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.532176 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.532102    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534396    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59e95e83-4c02-488c-8f19-917b84184cca-config-out\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-web-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534583    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-config-volume\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.534912 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.534842    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.535220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.535203    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59e95e83-4c02-488c-8f19-917b84184cca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.536263 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.536245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59e95e83-4c02-488c-8f19-917b84184cca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.544575 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.544558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65jh\" (UniqueName: \"kubernetes.io/projected/59e95e83-4c02-488c-8f19-917b84184cca-kube-api-access-x65jh\") pod \"alertmanager-main-0\" (UID: \"59e95e83-4c02-488c-8f19-917b84184cca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.642991 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.642968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:27:08.946438 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:08.946113    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:27:08.948684 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:27:08.948658    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e95e83_4c02_488c_8f19_917b84184cca.slice/crio-a951cbec2678ff98d206e108072c3da7660a97035a7f6719b73d6c6ae889b722 WatchSource:0}: Error finding container a951cbec2678ff98d206e108072c3da7660a97035a7f6719b73d6c6ae889b722: Status 404 returned error can't find the container with id a951cbec2678ff98d206e108072c3da7660a97035a7f6719b73d6c6ae889b722
Sep 19 08:27:09.265223 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:09.265198    2584 generic.go:356] "Generic (PLEG): container finished" podID="59e95e83-4c02-488c-8f19-917b84184cca" containerID="ada85fc97dbc46916b18a81d13aab37ab58676416180d73d335612ebb2a72644" exitCode=0
Sep 19 08:27:09.265325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:09.265262    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerDied","Data":"ada85fc97dbc46916b18a81d13aab37ab58676416180d73d335612ebb2a72644"}
Sep 19 08:27:09.265325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:09.265313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"a951cbec2678ff98d206e108072c3da7660a97035a7f6719b73d6c6ae889b722"}
Sep 19 08:27:09.906616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:09.906544    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1e31fa-5433-40a5-86b0-706fb8e12b76" path="/var/lib/kubelet/pods/ee1e31fa-5433-40a5-86b0-706fb8e12b76/volumes"
Sep 19 08:27:10.272671 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"1f7aa5f359c810c6ddc32580b5f1b79a10af337a4e6373955debec992d3faaa2"}
Sep 19 08:27:10.273046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"f121db6eb93ca9e94d00c4f47db53e450295206af66e0fa138a2549329dfc6fe"}
Sep 19 08:27:10.273046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272732    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"9a2c9ef73aee1596bf58bc88a5d5059a14ac4df869dcb44f155d7c5f393b8a05"}
Sep 19 08:27:10.273046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"904b49204b57f8ede906357f47fa630a8213038dfd8b5edcbf4565cd18573104"}
Sep 19 08:27:10.273046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"f9cacc23987d91f6374eee40903f4a8ab47924ac2e2306e472da68b14494c6b9"}
Sep 19 08:27:10.273046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.272806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59e95e83-4c02-488c-8f19-917b84184cca","Type":"ContainerStarted","Data":"ecd4ff55665609c3a635217e129d65c080205c357c6577b2bea622968c2af65c"}
Sep 19 08:27:10.313103 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.313047    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2959924210000002 podStartE2EDuration="2.313033021s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.0406ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:10.306335995 +0000 UTC m=+398.390836800" watchObservedRunningTime="2026-09-19 08:27:10.313033021 +0000 UTC m=+398.397533824"
Sep 19 08:27:10.652086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.651998    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69fcf6478b-r5n97"]
Sep 19 08:27:10.654750 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.654734    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.657491 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.657464    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 19 08:27:10.657652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.657574    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-77h6b\""
Sep 19 08:27:10.657737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.657714    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 19 08:27:10.657864 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.657847    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 19 08:27:10.658186 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.658166    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 19 08:27:10.658555 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.658536    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 19 08:27:10.663503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.663483    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 19 08:27:10.681409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.681388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69fcf6478b-r5n97"]
Sep 19 08:27:10.747312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-metrics-client-ca\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-serving-certs-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-federate-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zw88\" (UniqueName: \"kubernetes.io/projected/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-kube-api-access-2zw88\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.747616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.747601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848379 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848591 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-metrics-client-ca\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848645 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848703 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-serving-certs-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-federate-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.848756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.848749    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zw88\" (UniqueName: \"kubernetes.io/projected/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-kube-api-access-2zw88\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.849371 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.849341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-metrics-client-ca\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.849525 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.849507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.849525 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.849511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-serving-certs-ca-bundle\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.851219 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.851190    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.851331 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.851257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-federate-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.851400 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.851376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.851460 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.851423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-telemeter-client-tls\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.863216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.863193    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zw88\" (UniqueName: \"kubernetes.io/projected/2e5dba9b-6ee0-4868-b8cc-f01553e9af2d-kube-api-access-2zw88\") pod \"telemeter-client-69fcf6478b-r5n97\" (UID: \"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d\") " pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:10.965389 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:10.965370    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97"
Sep 19 08:27:11.265931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:11.265904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69fcf6478b-r5n97"]
Sep 19 08:27:11.266853 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:27:11.266828    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5dba9b_6ee0_4868_b8cc_f01553e9af2d.slice/crio-cd82649eb22f5a76013e52d8ad7737d8bfe7fd316e47d0eadf60bcff9aa801fb WatchSource:0}: Error finding container cd82649eb22f5a76013e52d8ad7737d8bfe7fd316e47d0eadf60bcff9aa801fb: Status 404 returned error can't find the container with id cd82649eb22f5a76013e52d8ad7737d8bfe7fd316e47d0eadf60bcff9aa801fb
Sep 19 08:27:11.278473 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:11.278449    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97" event={"ID":"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d","Type":"ContainerStarted","Data":"cd82649eb22f5a76013e52d8ad7737d8bfe7fd316e47d0eadf60bcff9aa801fb"}
Sep 19 08:27:13.290303 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.290270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97" event={"ID":"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d","Type":"ContainerStarted","Data":"4bce8afc21d19df49fee9d9e01eee9ba719d8840b77c3f5c10c1c5222900726f"}
Sep 19 08:27:13.290669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.290334    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97" event={"ID":"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d","Type":"ContainerStarted","Data":"1311a8a9fb799ddb319c24c1e2d43042af4abaa7d03a2df041aafe6d3de9b279"}
Sep 19 08:27:13.290669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.290358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97" event={"ID":"2e5dba9b-6ee0-4868-b8cc-f01553e9af2d","Type":"ContainerStarted","Data":"4628284baeecb99e34e4d8d55b043d27073d534bbca7cf0a85de09dfa173542a"}
Sep 19 08:27:13.319465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.319426    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69fcf6478b-r5n97" podStartSLOduration=1.48545025 podStartE2EDuration="3.319412126s" totalImagesPullingTime="1.833961876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:13.317339791 +0000 UTC m=+401.401840595" watchObservedRunningTime="2026-09-19 08:27:13.319412126 +0000 UTC m=+401.403912930"
Sep 19 08:27:13.416795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.416720    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:27:13.419032 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.419010    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.438373 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.438351    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:27:13.471943 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.471911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472061 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.471972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472061 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.472029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472061 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.472053    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbzs\" (UniqueName: \"kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.472108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.472151    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.472225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.472173    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573219 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573455 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbzs\" (UniqueName: \"kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573455 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573343    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.573455 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.573378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.574113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.574091    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.574223 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.574146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.574223 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.574195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.574344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.574280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.575849 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.575829    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.575928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.575904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.588482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.588459    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbzs\" (UniqueName: \"kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs\") pod \"console-599d75f558-dgrtc\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") " pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:13.729897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:13.729877    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:14.011395 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:14.011372    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:27:14.013507 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:27:14.013477    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0a1f09_a085_4e2f_a4e4_b7d45af15519.slice/crio-7b8291aea88a2b73c3dfb528ac4af0d73416c7fd574d5af70c11894ebc2a988e WatchSource:0}: Error finding container 7b8291aea88a2b73c3dfb528ac4af0d73416c7fd574d5af70c11894ebc2a988e: Status 404 returned error can't find the container with id 7b8291aea88a2b73c3dfb528ac4af0d73416c7fd574d5af70c11894ebc2a988e
Sep 19 08:27:14.296193 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:14.296112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d75f558-dgrtc" event={"ID":"0f0a1f09-a085-4e2f-a4e4-b7d45af15519","Type":"ContainerStarted","Data":"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"}
Sep 19 08:27:14.296193 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:14.296167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d75f558-dgrtc" event={"ID":"0f0a1f09-a085-4e2f-a4e4-b7d45af15519","Type":"ContainerStarted","Data":"7b8291aea88a2b73c3dfb528ac4af0d73416c7fd574d5af70c11894ebc2a988e"}
Sep 19 08:27:14.316169 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:14.316124    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599d75f558-dgrtc" podStartSLOduration=1.316109363 podStartE2EDuration="1.316109363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:14.314643595 +0000 UTC m=+402.399144397" watchObservedRunningTime="2026-09-19 08:27:14.316109363 +0000 UTC m=+402.400610166"
Sep 19 08:27:16.705626 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:16.705580    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" podUID="8e683a50-2d02-4b9b-b0cb-8744e2790ef9"
Sep 19 08:27:16.705626 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:16.705615    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vz85" podUID="6da27a61-af20-4407-934d-eda26bccb1b6"
Sep 19 08:27:17.309926 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:17.309888    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:27:17.309926 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:17.309913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:19.223460 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.223421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:27:19.226016 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.225974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e683a50-2d02-4b9b-b0cb-8744e2790ef9-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4tts9\" (UID: \"8e683a50-2d02-4b9b-b0cb-8744e2790ef9\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:27:19.324100 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.324071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:19.326297 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.326264    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6da27a61-af20-4407-934d-eda26bccb1b6-metrics-tls\") pod \"dns-default-4vz85\" (UID: \"6da27a61-af20-4407-934d-eda26bccb1b6\") " pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:19.414227 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.414203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dzp9\""
Sep 19 08:27:19.414307 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.414203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jbjhg\""
Sep 19 08:27:19.421211 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.421194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:19.421265 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.421212    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"
Sep 19 08:27:19.701589 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.701565    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vz85"]
Sep 19 08:27:19.704307 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:19.704287    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4tts9"]
Sep 19 08:27:19.704703 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:27:19.704685    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da27a61_af20_4407_934d_eda26bccb1b6.slice/crio-8563186f4ea37d0e00ceb5649ca62b528da2c5de6f2a8c10f3867021954c4e39 WatchSource:0}: Error finding container 8563186f4ea37d0e00ceb5649ca62b528da2c5de6f2a8c10f3867021954c4e39: Status 404 returned error can't find the container with id 8563186f4ea37d0e00ceb5649ca62b528da2c5de6f2a8c10f3867021954c4e39
Sep 19 08:27:19.706276 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:27:19.706255    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e683a50_2d02_4b9b_b0cb_8744e2790ef9.slice/crio-efb837ef2e098ed935ecf62738b0b23daea70f3047dda47b6b1613f15f0a092c WatchSource:0}: Error finding container efb837ef2e098ed935ecf62738b0b23daea70f3047dda47b6b1613f15f0a092c: Status 404 returned error can't find the container with id efb837ef2e098ed935ecf62738b0b23daea70f3047dda47b6b1613f15f0a092c
Sep 19 08:27:20.326326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:20.326253    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" event={"ID":"8e683a50-2d02-4b9b-b0cb-8744e2790ef9","Type":"ContainerStarted","Data":"efb837ef2e098ed935ecf62738b0b23daea70f3047dda47b6b1613f15f0a092c"}
Sep 19 08:27:20.327898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:20.327868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vz85" event={"ID":"6da27a61-af20-4407-934d-eda26bccb1b6","Type":"ContainerStarted","Data":"8563186f4ea37d0e00ceb5649ca62b528da2c5de6f2a8c10f3867021954c4e39"}
Sep 19 08:27:21.334715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:21.334667    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" event={"ID":"8e683a50-2d02-4b9b-b0cb-8744e2790ef9","Type":"ContainerStarted","Data":"a694cbb55d7a1de11bd42431f6ddcf0b79b56856e45003fd6d889cad02c334c3"}
Sep 19 08:27:21.353887 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:21.353827    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4tts9" podStartSLOduration=400.263454186 podStartE2EDuration="6m41.353807394s" totalImagesPullingTime="1.090353208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:21.352419747 +0000 UTC m=+409.436920551" watchObservedRunningTime="2026-09-19 08:27:21.353807394 +0000 UTC m=+409.438308199"
Sep 19 08:27:22.346727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:22.346687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vz85" event={"ID":"6da27a61-af20-4407-934d-eda26bccb1b6","Type":"ContainerStarted","Data":"bbb538fa213b86d22eb32be65cd816183c020a4176f86e861c4336e280acbc52"}
Sep 19 08:27:22.347124 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:22.346780    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vz85" event={"ID":"6da27a61-af20-4407-934d-eda26bccb1b6","Type":"ContainerStarted","Data":"ee23a116c8a93b3b67c63fbb258e1ac6489e4cda846e54723289bed931aed461"}
Sep 19 08:27:22.347124 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:22.346823    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:22.362855 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:22.362805    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4vz85" podStartSLOduration=373.637063857 podStartE2EDuration="6m15.362791017s" totalImagesPullingTime="1.72572716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:22.362307614 +0000 UTC m=+410.446808433" watchObservedRunningTime="2026-09-19 08:27:22.362791017 +0000 UTC m=+410.447291820"
Sep 19 08:27:23.730239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:23.730204    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:23.730639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:23.730261    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:23.734996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:23.734960    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:24.360480 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:24.360446    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:27:24.402472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:24.402438    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:27:32.354035 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:32.354002    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4vz85"
Sep 19 08:27:49.426735 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.426631    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b54fd5f79-p59wt" podUID="954752bb-b362-465e-a8e5-9a6547ed087f" containerName="console" containerID="cri-o://b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0" gracePeriod=15
Sep 19 08:27:49.666332 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.666310    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:27:49.745802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.745776    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkk2c\" (UniqueName: \"kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.745911 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.745815    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.745911 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.745857    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.746026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.745908    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.746026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.746128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746031    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.746128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746080    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca\") pod \"954752bb-b362-465e-a8e5-9a6547ed087f\" (UID: \"954752bb-b362-465e-a8e5-9a6547ed087f\") "
Sep 19 08:27:49.746381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746345    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:49.746590 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746554    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:49.746713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746592    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config" (OuterVolumeSpecName: "console-config") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:49.746713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.746608    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca" (OuterVolumeSpecName: "service-ca") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:49.747973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.747937    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c" (OuterVolumeSpecName: "kube-api-access-lkk2c") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "kube-api-access-lkk2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:49.748092 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.748015    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:49.748092 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.748030    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "954752bb-b362-465e-a8e5-9a6547ed087f" (UID: "954752bb-b362-465e-a8e5-9a6547ed087f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:49.847478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847455    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkk2c\" (UniqueName: \"kubernetes.io/projected/954752bb-b362-465e-a8e5-9a6547ed087f-kube-api-access-lkk2c\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847476    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847486    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-oauth-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847494    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/954752bb-b362-465e-a8e5-9a6547ed087f-console-oauth-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847502    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-console-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847510    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-trusted-ca-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:49.847597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:49.847518    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954752bb-b362-465e-a8e5-9a6547ed087f-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:27:50.473148 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.473109    2584 generic.go:356] "Generic (PLEG): container finished" podID="954752bb-b362-465e-a8e5-9a6547ed087f" containerID="b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0" exitCode=0
Sep 19 08:27:50.473632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.473163    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b54fd5f79-p59wt"
Sep 19 08:27:50.473632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.473165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b54fd5f79-p59wt" event={"ID":"954752bb-b362-465e-a8e5-9a6547ed087f","Type":"ContainerDied","Data":"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"}
Sep 19 08:27:50.473632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.473297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b54fd5f79-p59wt" event={"ID":"954752bb-b362-465e-a8e5-9a6547ed087f","Type":"ContainerDied","Data":"c560d353404910cff15dc0f0e690bec5c0b155171fc619345649c58e1477a37b"}
Sep 19 08:27:50.473632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.473324    2584 scope.go:117] "RemoveContainer" containerID="b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"
Sep 19 08:27:50.482053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.482032    2584 scope.go:117] "RemoveContainer" containerID="b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"
Sep 19 08:27:50.482382 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:27:50.482341    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0\": container with ID starting with b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0 not found: ID does not exist" containerID="b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"
Sep 19 08:27:50.482453 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.482399    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0"} err="failed to get container status \"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0\": rpc error: code = NotFound desc = could not find container \"b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0\": container with ID starting with b14b1e58e4ed423aca0b8e423a2e42bc47557119632ca39855c509d1191bb9e0 not found: ID does not exist"
Sep 19 08:27:50.492070 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.492045    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:27:50.498756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:50.498732    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b54fd5f79-p59wt"]
Sep 19 08:27:51.906820 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:27:51.906778    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954752bb-b362-465e-a8e5-9a6547ed087f" path="/var/lib/kubelet/pods/954752bb-b362-465e-a8e5-9a6547ed087f/volumes"
Sep 19 08:28:24.036775 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.036747    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:28:24.037267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.037126    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954752bb-b362-465e-a8e5-9a6547ed087f" containerName="console"
Sep 19 08:28:24.037267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.037141    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="954752bb-b362-465e-a8e5-9a6547ed087f" containerName="console"
Sep 19 08:28:24.037267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.037199    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="954752bb-b362-465e-a8e5-9a6547ed087f" containerName="console"
Sep 19 08:28:24.040356 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.040337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.051885 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.051862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:28:24.103720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103697    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.103855 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103728    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.103855 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.103855 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.104009 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzmh\" (UniqueName: \"kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.104009 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.104009 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.103952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.204366 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204336    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.204483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204381    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.204483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204407    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.204573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.204626 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzmh\" (UniqueName: \"kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.204855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.205208    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.205251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205478 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.205312    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.205590 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.205531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.207016 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.206977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.207094 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.207021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.213278 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.213255    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzmh\" (UniqueName: \"kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh\") pod \"console-78bd4d6b7b-ww9ms\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") " pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.351693 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.351625    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:24.622668 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:24.622523    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:28:24.625315 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:28:24.625286    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb538c54_627f_4097_b75b_4b7ddfc3b281.slice/crio-93d50171f08d1cf397d4d4c54784bac6ca5b3c995cf6d3713b0d0fff294bb3f8 WatchSource:0}: Error finding container 93d50171f08d1cf397d4d4c54784bac6ca5b3c995cf6d3713b0d0fff294bb3f8: Status 404 returned error can't find the container with id 93d50171f08d1cf397d4d4c54784bac6ca5b3c995cf6d3713b0d0fff294bb3f8
Sep 19 08:28:25.622433 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:25.622392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bd4d6b7b-ww9ms" event={"ID":"cb538c54-627f-4097-b75b-4b7ddfc3b281","Type":"ContainerStarted","Data":"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"}
Sep 19 08:28:25.622866 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:25.622451    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bd4d6b7b-ww9ms" event={"ID":"cb538c54-627f-4097-b75b-4b7ddfc3b281","Type":"ContainerStarted","Data":"93d50171f08d1cf397d4d4c54784bac6ca5b3c995cf6d3713b0d0fff294bb3f8"}
Sep 19 08:28:25.640433 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:25.640382    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78bd4d6b7b-ww9ms" podStartSLOduration=1.640363817 podStartE2EDuration="1.640363817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:25.638696506 +0000 UTC m=+473.723197310" watchObservedRunningTime="2026-09-19 08:28:25.640363817 +0000 UTC m=+473.724864619"
Sep 19 08:28:34.352015 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:34.351970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:34.352359 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:34.352104    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:34.357466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:34.357428    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:34.665784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:34.665711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:28:34.712430 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:34.712401    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:28:59.736916 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:59.736861    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599d75f558-dgrtc" podUID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" containerName="console" containerID="cri-o://f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2" gracePeriod=15
Sep 19 08:28:59.982329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:28:59.982307    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:29:00.073344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073281    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073319    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjbzs\" (UniqueName: \"kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073338    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073359    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073428    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073446    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073473    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert\") pod \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\" (UID: \"0f0a1f09-a085-4e2f-a4e4-b7d45af15519\") "
Sep 19 08:29:00.073755 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073648    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:29:00.073928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:29:00.073928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.073908    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca" (OuterVolumeSpecName: "service-ca") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:29:00.074093 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.074016    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config" (OuterVolumeSpecName: "console-config") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:29:00.075443 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.075409    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:29:00.075535 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.075462    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs" (OuterVolumeSpecName: "kube-api-access-cjbzs") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "kube-api-access-cjbzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:29:00.075535 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.075508    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0f0a1f09-a085-4e2f-a4e4-b7d45af15519" (UID: "0f0a1f09-a085-4e2f-a4e4-b7d45af15519"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:29:00.173966 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.173942    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-oauth-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.173966 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.173964    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-trusted-ca-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.174101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.173973    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.174101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.174002    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-oauth-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.174101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.174011    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbzs\" (UniqueName: \"kubernetes.io/projected/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-kube-api-access-cjbzs\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.174101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.174020    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.174101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.174028    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f0a1f09-a085-4e2f-a4e4-b7d45af15519-console-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:00.768150 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.768116    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" containerID="f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2" exitCode=0
Sep 19 08:29:00.768794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.768179    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d75f558-dgrtc"
Sep 19 08:29:00.768794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.768207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d75f558-dgrtc" event={"ID":"0f0a1f09-a085-4e2f-a4e4-b7d45af15519","Type":"ContainerDied","Data":"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"}
Sep 19 08:29:00.768794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.768281    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d75f558-dgrtc" event={"ID":"0f0a1f09-a085-4e2f-a4e4-b7d45af15519","Type":"ContainerDied","Data":"7b8291aea88a2b73c3dfb528ac4af0d73416c7fd574d5af70c11894ebc2a988e"}
Sep 19 08:29:00.768794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.768314    2584 scope.go:117] "RemoveContainer" containerID="f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"
Sep 19 08:29:00.778542 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.778524    2584 scope.go:117] "RemoveContainer" containerID="f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"
Sep 19 08:29:00.778803 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:29:00.778783    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2\": container with ID starting with f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2 not found: ID does not exist" containerID="f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"
Sep 19 08:29:00.778863 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.778811    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2"} err="failed to get container status \"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2\": rpc error: code = NotFound desc = could not find container \"f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2\": container with ID starting with f355aeab80886e3b30b76b19fd3bc745781b51f71c4c3035a95c8e51f0e52fa2 not found: ID does not exist"
Sep 19 08:29:00.789398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.789373    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:29:00.793308 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:00.793286    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599d75f558-dgrtc"]
Sep 19 08:29:01.907531 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:01.907493    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" path="/var/lib/kubelet/pods/0f0a1f09-a085-4e2f-a4e4-b7d45af15519/volumes"
Sep 19 08:29:35.202027 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.201994    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"]
Sep 19 08:29:35.202409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.202313    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" containerName="console"
Sep 19 08:29:35.202409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.202323    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" containerName="console"
Sep 19 08:29:35.202409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.202383    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0a1f09-a085-4e2f-a4e4-b7d45af15519" containerName="console"
Sep 19 08:29:35.206352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.206337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.209319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.209289    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 08:29:35.209453 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.209353    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-57sjz\""
Sep 19 08:29:35.210162 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.210144    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 08:29:35.215315 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.215293    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"]
Sep 19 08:29:35.222744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.222723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.222840 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.222748    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.222902 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.222864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ccn\" (UniqueName: \"kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.323142 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.323121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.323242 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.323144    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.323242 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.323192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ccn\" (UniqueName: \"kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.323486 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.323465    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.323560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.323529    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.331365 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.331347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ccn\" (UniqueName: \"kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.516143 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.516123    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:35.783835 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.783760    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"]
Sep 19 08:29:35.786687 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:29:35.786660    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f637a1f_5fad_4ed4_b8c0_13bedb435e58.slice/crio-b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780 WatchSource:0}: Error finding container b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780: Status 404 returned error can't find the container with id b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780
Sep 19 08:29:35.923784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:35.923760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx" event={"ID":"9f637a1f-5fad-4ed4-b8c0-13bedb435e58","Type":"ContainerStarted","Data":"b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780"}
Sep 19 08:29:41.951538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:41.951507    2584 generic.go:356] "Generic (PLEG): container finished" podID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerID="17c9c76d578b0411e89dc5e7d819687b831faeca8354e7a291e11fd7b0468fda" exitCode=0
Sep 19 08:29:41.951940 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:41.951591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx" event={"ID":"9f637a1f-5fad-4ed4-b8c0-13bedb435e58","Type":"ContainerDied","Data":"17c9c76d578b0411e89dc5e7d819687b831faeca8354e7a291e11fd7b0468fda"}
Sep 19 08:29:44.966423 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:44.966389    2584 generic.go:356] "Generic (PLEG): container finished" podID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerID="5d556af0baf5b1add76dc62b1b95e31ec6b195ad842be0e26778cbcb4f44e807" exitCode=0
Sep 19 08:29:44.966876 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:44.966475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx" event={"ID":"9f637a1f-5fad-4ed4-b8c0-13bedb435e58","Type":"ContainerDied","Data":"5d556af0baf5b1add76dc62b1b95e31ec6b195ad842be0e26778cbcb4f44e807"}
Sep 19 08:29:51.999804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:51.999775    2584 generic.go:356] "Generic (PLEG): container finished" podID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerID="417acb14216f04546a0a1b08fdc79535161b956a2fcbe95bfa181cc25f86bc3c" exitCode=0
Sep 19 08:29:52.000128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:51.999845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx" event={"ID":"9f637a1f-5fad-4ed4-b8c0-13bedb435e58","Type":"ContainerDied","Data":"417acb14216f04546a0a1b08fdc79535161b956a2fcbe95bfa181cc25f86bc3c"}
Sep 19 08:29:53.126253 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.126231    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:53.171261 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.171239    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util\") pod \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") "
Sep 19 08:29:53.171358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.171291    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle\") pod \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") "
Sep 19 08:29:53.171358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.171349    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7ccn\" (UniqueName: \"kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn\") pod \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\" (UID: \"9f637a1f-5fad-4ed4-b8c0-13bedb435e58\") "
Sep 19 08:29:53.171930 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.171898    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle" (OuterVolumeSpecName: "bundle") pod "9f637a1f-5fad-4ed4-b8c0-13bedb435e58" (UID: "9f637a1f-5fad-4ed4-b8c0-13bedb435e58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:29:53.173476 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.173452    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn" (OuterVolumeSpecName: "kube-api-access-n7ccn") pod "9f637a1f-5fad-4ed4-b8c0-13bedb435e58" (UID: "9f637a1f-5fad-4ed4-b8c0-13bedb435e58"). InnerVolumeSpecName "kube-api-access-n7ccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:29:53.175459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.175439    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util" (OuterVolumeSpecName: "util") pod "9f637a1f-5fad-4ed4-b8c0-13bedb435e58" (UID: "9f637a1f-5fad-4ed4-b8c0-13bedb435e58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:29:53.272305 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.272251    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:53.272305 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.272280    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7ccn\" (UniqueName: \"kubernetes.io/projected/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-kube-api-access-n7ccn\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:53.272305 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:53.272291    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f637a1f-5fad-4ed4-b8c0-13bedb435e58-util\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:29:54.009231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:54.009206    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx"
Sep 19 08:29:54.009231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:54.009217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a987mx" event={"ID":"9f637a1f-5fad-4ed4-b8c0-13bedb435e58","Type":"ContainerDied","Data":"b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780"}
Sep 19 08:29:54.009388 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:54.009254    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7bd432a30c2c2f46655c6e7d438aa442d35e4b93a77a3417b0d68f8a9d42780"
Sep 19 08:29:58.116392 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116354    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"]
Sep 19 08:29:58.116837 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116819    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="util"
Sep 19 08:29:58.116904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116841    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="util"
Sep 19 08:29:58.116904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116874    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="extract"
Sep 19 08:29:58.116904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116883    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="extract"
Sep 19 08:29:58.116904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116894    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="pull"
Sep 19 08:29:58.116904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116903    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="pull"
Sep 19 08:29:58.117175 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.116976    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f637a1f-5fad-4ed4-b8c0-13bedb435e58" containerName="extract"
Sep 19 08:29:58.121215 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.121194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.123836 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.123817    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 19 08:29:58.123994 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.123973    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 19 08:29:58.124161 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.124146    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jb62l\""
Sep 19 08:29:58.124362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.124351    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 19 08:29:58.131376 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.131352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"]
Sep 19 08:29:58.207186 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.207161    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsmq\" (UniqueName: \"kubernetes.io/projected/72dcd608-9875-4479-a3cc-0f1618e1e8f9-kube-api-access-vdsmq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.207291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.207199    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/72dcd608-9875-4479-a3cc-0f1618e1e8f9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.308083 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.308056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsmq\" (UniqueName: \"kubernetes.io/projected/72dcd608-9875-4479-a3cc-0f1618e1e8f9-kube-api-access-vdsmq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.308173 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.308110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/72dcd608-9875-4479-a3cc-0f1618e1e8f9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.310352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.310326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/72dcd608-9875-4479-a3cc-0f1618e1e8f9-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.319111 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.319087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsmq\" (UniqueName: \"kubernetes.io/projected/72dcd608-9875-4479-a3cc-0f1618e1e8f9-kube-api-access-vdsmq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwsks\" (UID: \"72dcd608-9875-4479-a3cc-0f1618e1e8f9\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.431026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.430964    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:29:58.712127 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:58.712105    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"]
Sep 19 08:29:58.714879 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:29:58.714856    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dcd608_9875_4479_a3cc_0f1618e1e8f9.slice/crio-a39f61735d89ed2ccbcc36430f12da7ecaa8f77a86753ed34ddd76a932aa7bc6 WatchSource:0}: Error finding container a39f61735d89ed2ccbcc36430f12da7ecaa8f77a86753ed34ddd76a932aa7bc6: Status 404 returned error can't find the container with id a39f61735d89ed2ccbcc36430f12da7ecaa8f77a86753ed34ddd76a932aa7bc6
Sep 19 08:29:59.031525 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:29:59.031495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks" event={"ID":"72dcd608-9875-4479-a3cc-0f1618e1e8f9","Type":"ContainerStarted","Data":"a39f61735d89ed2ccbcc36430f12da7ecaa8f77a86753ed34ddd76a932aa7bc6"}
Sep 19 08:30:03.054285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.054255    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"]
Sep 19 08:30:03.056402 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.056385    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks" event={"ID":"72dcd608-9875-4479-a3cc-0f1618e1e8f9","Type":"ContainerStarted","Data":"8ddd8808f4cdb071dbc6919679bf238a05d915e044d5c41cfa8772cd3226c580"}
Sep 19 08:30:03.056477 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.056413    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:30:03.056514 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.056492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.059334 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.059274    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 19 08:30:03.059334 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.059302    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 19 08:30:03.059538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.059302    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ln47x\""
Sep 19 08:30:03.065429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.065406    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"]
Sep 19 08:30:03.075457 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.075384    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks" podStartSLOduration=1.748830152 podStartE2EDuration="5.075367359s" totalImagesPullingTime="3.326537207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:03.073827352 +0000 UTC m=+571.158328154" watchObservedRunningTime="2026-09-19 08:30:03.075367359 +0000 UTC m=+571.159868164"
Sep 19 08:30:03.149641 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.149615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgm5\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-kube-api-access-zxgm5\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.149789 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.149663    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d863cf9d-a234-4766-ae67-e518df7e0f50-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.149789 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.149742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.250665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.250641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d863cf9d-a234-4766-ae67-e518df7e0f50-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.250761 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.250676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.250761 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.250707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgm5\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-kube-api-access-zxgm5\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.250832 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.250804    2584 secret.go:281] references non-existent secret key: tls.crt
Sep 19 08:30:03.250832 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.250823    2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 19 08:30:03.250902 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.250843    2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22: references non-existent secret key: tls.crt
Sep 19 08:30:03.250935 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.250905    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates podName:d863cf9d-a234-4766-ae67-e518df7e0f50 nodeName:}" failed. No retries permitted until 2026-09-19 08:30:03.75088838 +0000 UTC m=+571.835389161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates") pod "keda-metrics-apiserver-77857b7b98-jnc22" (UID: "d863cf9d-a234-4766-ae67-e518df7e0f50") : references non-existent secret key: tls.crt
Sep 19 08:30:03.250999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.250967    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d863cf9d-a234-4766-ae67-e518df7e0f50-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.259159 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.259140    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgm5\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-kube-api-access-zxgm5\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.631930 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.631894    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4mh84"]
Sep 19 08:30:03.634239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.634226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.636691 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.636674    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 19 08:30:03.645973 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.645948    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4mh84"]
Sep 19 08:30:03.754676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.754651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-certificates\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.754811 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.754683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjtq\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-kube-api-access-htjtq\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.754811 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.754741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:03.754894 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.754875    2584 secret.go:281] references non-existent secret key: tls.crt
Sep 19 08:30:03.754930 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.754894    2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 19 08:30:03.754930 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.754914    2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22: references non-existent secret key: tls.crt
Sep 19 08:30:03.755012 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:30:03.754959    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates podName:d863cf9d-a234-4766-ae67-e518df7e0f50 nodeName:}" failed. No retries permitted until 2026-09-19 08:30:04.754946336 +0000 UTC m=+572.839447129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates") pod "keda-metrics-apiserver-77857b7b98-jnc22" (UID: "d863cf9d-a234-4766-ae67-e518df7e0f50") : references non-existent secret key: tls.crt
Sep 19 08:30:03.855251 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.855227    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-certificates\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.855388 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.855260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjtq\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-kube-api-access-htjtq\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.857476 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.857450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-certificates\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.862788 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.862768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjtq\" (UniqueName: \"kubernetes.io/projected/a85bdf37-718b-4f6f-81c2-1fd358c98430-kube-api-access-htjtq\") pod \"keda-admission-6c4d864b86-4mh84\" (UID: \"a85bdf37-718b-4f6f-81c2-1fd358c98430\") " pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:03.945306 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:03.945236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:04.214696 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:04.214676    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4mh84"]
Sep 19 08:30:04.217057 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:30:04.217031    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85bdf37_718b_4f6f_81c2_1fd358c98430.slice/crio-9ab12b59a67268bad9e6135d7f3ba5ef690130b92d760802e6e4216102479156 WatchSource:0}: Error finding container 9ab12b59a67268bad9e6135d7f3ba5ef690130b92d760802e6e4216102479156: Status 404 returned error can't find the container with id 9ab12b59a67268bad9e6135d7f3ba5ef690130b92d760802e6e4216102479156
Sep 19 08:30:04.764492 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:04.764459    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:04.766898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:04.766869    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d863cf9d-a234-4766-ae67-e518df7e0f50-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jnc22\" (UID: \"d863cf9d-a234-4766-ae67-e518df7e0f50\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:04.867775 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:04.867751    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:05.063815 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:05.063745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4mh84" event={"ID":"a85bdf37-718b-4f6f-81c2-1fd358c98430","Type":"ContainerStarted","Data":"9ab12b59a67268bad9e6135d7f3ba5ef690130b92d760802e6e4216102479156"}
Sep 19 08:30:05.135812 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:05.135791    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"]
Sep 19 08:30:05.137896 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:30:05.137866    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd863cf9d_a234_4766_ae67_e518df7e0f50.slice/crio-312a10a939db96c3a1c38718246425428415c5d73b162916f9d966d39ee07822 WatchSource:0}: Error finding container 312a10a939db96c3a1c38718246425428415c5d73b162916f9d966d39ee07822: Status 404 returned error can't find the container with id 312a10a939db96c3a1c38718246425428415c5d73b162916f9d966d39ee07822
Sep 19 08:30:06.068329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:06.068296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22" event={"ID":"d863cf9d-a234-4766-ae67-e518df7e0f50","Type":"ContainerStarted","Data":"312a10a939db96c3a1c38718246425428415c5d73b162916f9d966d39ee07822"}
Sep 19 08:30:08.078872 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:08.078834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4mh84" event={"ID":"a85bdf37-718b-4f6f-81c2-1fd358c98430","Type":"ContainerStarted","Data":"fd93bcc4b47aad951269da121cd353e94843e748355bc6d315cf336aa1b5dbaf"}
Sep 19 08:30:08.079238 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:08.078900    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:08.095774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:08.095727    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-4mh84" podStartSLOduration=1.9462179179999999 podStartE2EDuration="5.095710041s" totalImagesPullingTime="3.149492123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:08.093729886 +0000 UTC m=+576.178230689" watchObservedRunningTime="2026-09-19 08:30:08.095710041 +0000 UTC m=+576.180210844"
Sep 19 08:30:12.098650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:12.098570    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22" event={"ID":"d863cf9d-a234-4766-ae67-e518df7e0f50","Type":"ContainerStarted","Data":"5717749cb029e4b999ce94520b8c45210adbc0ad1c81e38d7e4ed94824ec5ef8"}
Sep 19 08:30:12.098650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:12.098645    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:12.116851 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:12.116798    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22" podStartSLOduration=2.47178003 podStartE2EDuration="9.116782423s" totalImagesPullingTime="6.645002393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:12.114825465 +0000 UTC m=+580.199326267" watchObservedRunningTime="2026-09-19 08:30:12.116782423 +0000 UTC m=+580.201283224"
Sep 19 08:30:23.108059 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:23.108031    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jnc22"
Sep 19 08:30:24.060907 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:24.060874    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwsks"
Sep 19 08:30:29.085617 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:29.085586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-4mh84"
Sep 19 08:30:33.887517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:33.887489    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:30:33.887919 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:33.887615    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:30:33.892636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:33.892615    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:30:33.893035 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:30:33.893016    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:31:19.834394 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.834365    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5bpfd"]
Sep 19 08:31:19.836646 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.836621    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.839039 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.839018    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:31:19.839159 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.839066    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:31:19.839812 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.839795    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9trcl\""
Sep 19 08:31:19.839928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.839830    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:31:19.852649 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.852627    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5bpfd"]
Sep 19 08:31:19.873681 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.873662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcskh\" (UniqueName: \"kubernetes.io/projected/74811801-cb1a-4466-8962-8d0513ed173a-kube-api-access-jcskh\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.873801 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.873756    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74811801-cb1a-4466-8962-8d0513ed173a-data\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.974077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.974056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74811801-cb1a-4466-8962-8d0513ed173a-data\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.974184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.974120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcskh\" (UniqueName: \"kubernetes.io/projected/74811801-cb1a-4466-8962-8d0513ed173a-kube-api-access-jcskh\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.974749 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.974726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/74811801-cb1a-4466-8962-8d0513ed173a-data\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:19.984236 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:19.984209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcskh\" (UniqueName: \"kubernetes.io/projected/74811801-cb1a-4466-8962-8d0513ed173a-kube-api-access-jcskh\") pod \"seaweedfs-64568bcd49-5bpfd\" (UID: \"74811801-cb1a-4466-8962-8d0513ed173a\") " pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:20.152759 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:20.152693    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:20.419762 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:20.419681    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5bpfd"]
Sep 19 08:31:20.424327 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:31:20.424297    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74811801_cb1a_4466_8962_8d0513ed173a.slice/crio-e67f2b34cfca475c677243d1fa36189d146b3462ac19937ca0319191509edabd WatchSource:0}: Error finding container e67f2b34cfca475c677243d1fa36189d146b3462ac19937ca0319191509edabd: Status 404 returned error can't find the container with id e67f2b34cfca475c677243d1fa36189d146b3462ac19937ca0319191509edabd
Sep 19 08:31:20.425687 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:20.425663    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:31:21.420722 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:21.420687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5bpfd" event={"ID":"74811801-cb1a-4466-8962-8d0513ed173a","Type":"ContainerStarted","Data":"e67f2b34cfca475c677243d1fa36189d146b3462ac19937ca0319191509edabd"}
Sep 19 08:31:23.432616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:23.432537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5bpfd" event={"ID":"74811801-cb1a-4466-8962-8d0513ed173a","Type":"ContainerStarted","Data":"fc7ceb27f4128001e491ed5d7796df896f0cecb43318d3ab27aee0abeab1b57f"}
Sep 19 08:31:23.432977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:23.432649    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:23.450604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:23.450561    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5bpfd" podStartSLOduration=1.90018745 podStartE2EDuration="4.45054826s" totalImagesPullingTime="2.55036081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:23.448309664 +0000 UTC m=+651.532810478" watchObservedRunningTime="2026-09-19 08:31:23.45054826 +0000 UTC m=+651.535049062"
Sep 19 08:31:29.439952 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:29.439920    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5bpfd"
Sep 19 08:31:40.414206 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.414178    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6df54556d4-wrkns"]
Sep 19 08:31:40.419551 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.419530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.428772 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.428748    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df54556d4-wrkns"]
Sep 19 08:31:40.528297 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-oauth-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528442 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-trusted-ca-bundle\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528442 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528369    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-oauth-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528442 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-console-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528437    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528480    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwljv\" (UniqueName: \"kubernetes.io/projected/420a2a27-bc57-454f-b61e-c0305b77e891-kube-api-access-bwljv\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.528545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.528515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-service-ca\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629404    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-trusted-ca-bundle\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629558 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-oauth-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629558 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-console-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629674 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629712    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwljv\" (UniqueName: \"kubernetes.io/projected/420a2a27-bc57-454f-b61e-c0305b77e891-kube-api-access-bwljv\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629788 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629773    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-service-ca\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.629843 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.629805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-oauth-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.630427 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.630406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-oauth-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.630565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.630451    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-console-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.630565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.630491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-service-ca\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.630565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.630534    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/420a2a27-bc57-454f-b61e-c0305b77e891-trusted-ca-bundle\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.632080 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.632052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-serving-cert\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.632080 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.632078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/420a2a27-bc57-454f-b61e-c0305b77e891-console-oauth-config\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.638358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.638340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwljv\" (UniqueName: \"kubernetes.io/projected/420a2a27-bc57-454f-b61e-c0305b77e891-kube-api-access-bwljv\") pod \"console-6df54556d4-wrkns\" (UID: \"420a2a27-bc57-454f-b61e-c0305b77e891\") " pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:40.730804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:40.730788    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:41.001435 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:41.001413    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6df54556d4-wrkns"]
Sep 19 08:31:41.003588 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:31:41.003552    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420a2a27_bc57_454f_b61e_c0305b77e891.slice/crio-bd8611fbef39cc6edf55a1a04a0ab86c23e7639b5adcd879b59c0af98ffe73f0 WatchSource:0}: Error finding container bd8611fbef39cc6edf55a1a04a0ab86c23e7639b5adcd879b59c0af98ffe73f0: Status 404 returned error can't find the container with id bd8611fbef39cc6edf55a1a04a0ab86c23e7639b5adcd879b59c0af98ffe73f0
Sep 19 08:31:41.520203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:41.520165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df54556d4-wrkns" event={"ID":"420a2a27-bc57-454f-b61e-c0305b77e891","Type":"ContainerStarted","Data":"1edc6ce2afcbe77ad150858a5d34e49311d10eb04c257928940f352f3e9713ed"}
Sep 19 08:31:41.520654 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:41.520222    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6df54556d4-wrkns" event={"ID":"420a2a27-bc57-454f-b61e-c0305b77e891","Type":"ContainerStarted","Data":"bd8611fbef39cc6edf55a1a04a0ab86c23e7639b5adcd879b59c0af98ffe73f0"}
Sep 19 08:31:50.731067 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:50.731031    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:50.731558 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:50.731090    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:50.736011 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:50.735969    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:50.757802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:50.757756    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6df54556d4-wrkns" podStartSLOduration=10.757740808 podStartE2EDuration="10.757740808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:41.536957586 +0000 UTC m=+669.621458388" watchObservedRunningTime="2026-09-19 08:31:50.757740808 +0000 UTC m=+678.842241610"
Sep 19 08:31:51.570296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:51.570271    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6df54556d4-wrkns"
Sep 19 08:31:51.614082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:31:51.614050    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:32:16.639400 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:16.639310    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78bd4d6b7b-ww9ms" podUID="cb538c54-627f-4097-b75b-4b7ddfc3b281" containerName="console" containerID="cri-o://d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c" gracePeriod=15
Sep 19 08:32:16.878530 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:16.878506    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:32:17.007636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007613    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007760 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007647    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007760 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007669    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007760 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007690    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvzmh\" (UniqueName: \"kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007760 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007748    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.007944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.007806    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca\") pod \"cb538c54-627f-4097-b75b-4b7ddfc3b281\" (UID: \"cb538c54-627f-4097-b75b-4b7ddfc3b281\") "
Sep 19 08:32:17.008122 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.008093    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config" (OuterVolumeSpecName: "console-config") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:32:17.008183 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.008128    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:32:17.008183 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.008168    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:32:17.008744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.008717    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca" (OuterVolumeSpecName: "service-ca") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:32:17.009811 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.009782    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:32:17.009916 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.009831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:32:17.009916 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.009851    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh" (OuterVolumeSpecName: "kube-api-access-gvzmh") pod "cb538c54-627f-4097-b75b-4b7ddfc3b281" (UID: "cb538c54-627f-4097-b75b-4b7ddfc3b281"). InnerVolumeSpecName "kube-api-access-gvzmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:32:17.109325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109302    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-trusted-ca-bundle\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109323    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-oauth-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109333    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-service-ca\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109341    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-oauth-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109349    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109357    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb538c54-627f-4097-b75b-4b7ddfc3b281-console-serving-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.109447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.109365    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvzmh\" (UniqueName: \"kubernetes.io/projected/cb538c54-627f-4097-b75b-4b7ddfc3b281-kube-api-access-gvzmh\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:32:17.691897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.691862    2584 generic.go:356] "Generic (PLEG): container finished" podID="cb538c54-627f-4097-b75b-4b7ddfc3b281" containerID="d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c" exitCode=0
Sep 19 08:32:17.692289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.691917    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bd4d6b7b-ww9ms"
Sep 19 08:32:17.692289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.691947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bd4d6b7b-ww9ms" event={"ID":"cb538c54-627f-4097-b75b-4b7ddfc3b281","Type":"ContainerDied","Data":"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"}
Sep 19 08:32:17.692289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.692011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bd4d6b7b-ww9ms" event={"ID":"cb538c54-627f-4097-b75b-4b7ddfc3b281","Type":"ContainerDied","Data":"93d50171f08d1cf397d4d4c54784bac6ca5b3c995cf6d3713b0d0fff294bb3f8"}
Sep 19 08:32:17.692289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.692036    2584 scope.go:117] "RemoveContainer" containerID="d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"
Sep 19 08:32:17.700883 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.700866    2584 scope.go:117] "RemoveContainer" containerID="d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"
Sep 19 08:32:17.701187 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:32:17.701164    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c\": container with ID starting with d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c not found: ID does not exist" containerID="d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"
Sep 19 08:32:17.701282 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.701193    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c"} err="failed to get container status \"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c\": rpc error: code = NotFound desc = could not find container \"d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c\": container with ID starting with d78eae85a592566de0d13ab24986777f1ce7bfa0626023a767cc11f637723f3c not found: ID does not exist"
Sep 19 08:32:17.713405 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.713384    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:32:17.715029 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.715012    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bd4d6b7b-ww9ms"]
Sep 19 08:32:17.906616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:17.906589    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb538c54-627f-4097-b75b-4b7ddfc3b281" path="/var/lib/kubelet/pods/cb538c54-627f-4097-b75b-4b7ddfc3b281/volumes"
Sep 19 08:32:37.161600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.161567    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-6xcvd"]
Sep 19 08:32:37.162098 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.162073    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb538c54-627f-4097-b75b-4b7ddfc3b281" containerName="console"
Sep 19 08:32:37.162098 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.162089    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb538c54-627f-4097-b75b-4b7ddfc3b281" containerName="console"
Sep 19 08:32:37.162208 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.162194    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb538c54-627f-4097-b75b-4b7ddfc3b281" containerName="console"
Sep 19 08:32:37.166697 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.166678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.169360 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.169335    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 19 08:32:37.169477 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.169399    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-w94js\""
Sep 19 08:32:37.176384 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.176364    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6xcvd"]
Sep 19 08:32:37.184296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.184275    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdm2c"]
Sep 19 08:32:37.187347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.187332    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.189940 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.189919    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 19 08:32:37.190052 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.189968    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-k942g\""
Sep 19 08:32:37.190124 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.189918    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 19 08:32:37.198355 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.198331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdm2c"]
Sep 19 08:32:37.251455 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.251431    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0424cbf1-f84f-4ec0-bbff-6095fffc8027-tls-certs\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.251551 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.251469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.251551 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.251494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lhd\" (UniqueName: \"kubernetes.io/projected/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-kube-api-access-k5lhd\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.251551 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.251517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwkc\" (UniqueName: \"kubernetes.io/projected/0424cbf1-f84f-4ec0-bbff-6095fffc8027-kube-api-access-5bwkc\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.251665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.251560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.352095 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.352068    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0424cbf1-f84f-4ec0-bbff-6095fffc8027-tls-certs\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.352199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.352114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.352199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.352151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lhd\" (UniqueName: \"kubernetes.io/projected/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-kube-api-access-k5lhd\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.352199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.352181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwkc\" (UniqueName: \"kubernetes.io/projected/0424cbf1-f84f-4ec0-bbff-6095fffc8027-kube-api-access-5bwkc\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.352352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.352225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.352352 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:32:37.352269    2584 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 19 08:32:37.352449 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:32:37.352361    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert podName:7f8f9c4b-1f32-4dbd-95dd-5a035308252a nodeName:}" failed. No retries permitted until 2026-09-19 08:32:37.852341334 +0000 UTC m=+725.936842134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert") pod "odh-model-controller-7b966d7984-hdm2c" (UID: "7f8f9c4b-1f32-4dbd-95dd-5a035308252a") : secret "odh-model-controller-webhook-cert" not found
Sep 19 08:32:37.354447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.354428    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0424cbf1-f84f-4ec0-bbff-6095fffc8027-tls-certs\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.354562 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.354545    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.360387 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.360358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwkc\" (UniqueName: \"kubernetes.io/projected/0424cbf1-f84f-4ec0-bbff-6095fffc8027-kube-api-access-5bwkc\") pod \"model-serving-api-7b954ffc95-6xcvd\" (UID: \"0424cbf1-f84f-4ec0-bbff-6095fffc8027\") " pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.360963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.360944    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lhd\" (UniqueName: \"kubernetes.io/projected/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-kube-api-access-k5lhd\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.479111 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.479091    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:37.749513 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.749492    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6xcvd"]
Sep 19 08:32:37.751036 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:32:37.751008    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0424cbf1_f84f_4ec0_bbff_6095fffc8027.slice/crio-fed45da422871aa92daf0436f9054262c1ca4bc92c530521cd68cfe91c373ba7 WatchSource:0}: Error finding container fed45da422871aa92daf0436f9054262c1ca4bc92c530521cd68cfe91c373ba7: Status 404 returned error can't find the container with id fed45da422871aa92daf0436f9054262c1ca4bc92c530521cd68cfe91c373ba7
Sep 19 08:32:37.791209 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.791182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6xcvd" event={"ID":"0424cbf1-f84f-4ec0-bbff-6095fffc8027","Type":"ContainerStarted","Data":"fed45da422871aa92daf0436f9054262c1ca4bc92c530521cd68cfe91c373ba7"}
Sep 19 08:32:37.856192 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.856167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:37.858412 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:37.858393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f8f9c4b-1f32-4dbd-95dd-5a035308252a-cert\") pod \"odh-model-controller-7b966d7984-hdm2c\" (UID: \"7f8f9c4b-1f32-4dbd-95dd-5a035308252a\") " pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:38.097822 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:38.097758    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:38.368283 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:38.368253    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdm2c"]
Sep 19 08:32:38.370931 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:32:38.370899    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8f9c4b_1f32_4dbd_95dd_5a035308252a.slice/crio-5ce0206eee97f4eeabb4764d820123a5c4356bba5b10f82271f4d3ae7bf2ad12 WatchSource:0}: Error finding container 5ce0206eee97f4eeabb4764d820123a5c4356bba5b10f82271f4d3ae7bf2ad12: Status 404 returned error can't find the container with id 5ce0206eee97f4eeabb4764d820123a5c4356bba5b10f82271f4d3ae7bf2ad12
Sep 19 08:32:38.798255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:38.798223    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hdm2c" event={"ID":"7f8f9c4b-1f32-4dbd-95dd-5a035308252a","Type":"ContainerStarted","Data":"5ce0206eee97f4eeabb4764d820123a5c4356bba5b10f82271f4d3ae7bf2ad12"}
Sep 19 08:32:41.816191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.816155    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6xcvd" event={"ID":"0424cbf1-f84f-4ec0-bbff-6095fffc8027","Type":"ContainerStarted","Data":"1090afaa1f4e414912db4556fb41ff7b9491fca8a23a7524ab99a39d3fb83f9c"}
Sep 19 08:32:41.816661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.816237    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:32:41.817760 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.817733    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hdm2c" event={"ID":"7f8f9c4b-1f32-4dbd-95dd-5a035308252a","Type":"ContainerStarted","Data":"5d83a53286cadc2646ab947a6d78adf4ec67a00f23d2aee7e26c86929f9d7a9c"}
Sep 19 08:32:41.817865 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.817780    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:41.832353 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.832306    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-6xcvd" podStartSLOduration=1.446572314 podStartE2EDuration="4.832293147s" totalImagesPullingTime="3.385720833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:41.83108069 +0000 UTC m=+729.915581492" watchObservedRunningTime="2026-09-19 08:32:41.832293147 +0000 UTC m=+729.916793948"
Sep 19 08:32:41.846602 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:41.846559    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-hdm2c" podStartSLOduration=2.034605014 podStartE2EDuration="4.846544432s" totalImagesPullingTime="2.811939418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:41.846369327 +0000 UTC m=+729.930870142" watchObservedRunningTime="2026-09-19 08:32:41.846544432 +0000 UTC m=+729.931045233"
Sep 19 08:32:52.825773 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:52.825742    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-hdm2c"
Sep 19 08:32:52.827650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:32:52.827631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-6xcvd"
Sep 19 08:34:11.631998 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.631963    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:11.635273 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.635258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.637570 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.637552    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 19 08:34:11.643077 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.643046    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:11.758711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.758689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjrl\" (UniqueName: \"kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.758810 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.758731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.859679 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.859657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.859794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.859719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjrl\" (UniqueName: \"kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.860089 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.860068    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.867956 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.867935    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjrl\" (UniqueName: \"kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl\") pod \"seaweedfs-tls-custom-6644bf5d6c-mmwtw\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:11.945358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:11.945307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:12.212466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:12.212437    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:12.214469 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:34:12.214438    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c664b71_d252_4d8a_9020_b5d4482d443f.slice/crio-1d4ac7a160e00c618f064aa512f8fe2c7da351bc4f66df3da9f962ca8d63ea56 WatchSource:0}: Error finding container 1d4ac7a160e00c618f064aa512f8fe2c7da351bc4f66df3da9f962ca8d63ea56: Status 404 returned error can't find the container with id 1d4ac7a160e00c618f064aa512f8fe2c7da351bc4f66df3da9f962ca8d63ea56
Sep 19 08:34:12.230748 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:12.230717    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" event={"ID":"0c664b71-d252-4d8a-9020-b5d4482d443f","Type":"ContainerStarted","Data":"1d4ac7a160e00c618f064aa512f8fe2c7da351bc4f66df3da9f962ca8d63ea56"}
Sep 19 08:34:13.239477 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:13.239446    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" event={"ID":"0c664b71-d252-4d8a-9020-b5d4482d443f","Type":"ContainerStarted","Data":"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"}
Sep 19 08:34:13.255482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:13.255438    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" podStartSLOduration=1.993809846 podStartE2EDuration="2.255424228s" totalImagesPullingTime="261.614382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:13.254258799 +0000 UTC m=+821.338759614" watchObservedRunningTime="2026-09-19 08:34:13.255424228 +0000 UTC m=+821.339925029"
Sep 19 08:34:14.121325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:14.121295    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:15.249145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:15.249098    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" podUID="0c664b71-d252-4d8a-9020-b5d4482d443f" containerName="seaweedfs-tls-custom" containerID="cri-o://28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49" gracePeriod=30
Sep 19 08:34:43.177705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.177681    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:43.290358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.290284    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data\") pod \"0c664b71-d252-4d8a-9020-b5d4482d443f\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") "
Sep 19 08:34:43.290502 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.290399    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbjrl\" (UniqueName: \"kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl\") pod \"0c664b71-d252-4d8a-9020-b5d4482d443f\" (UID: \"0c664b71-d252-4d8a-9020-b5d4482d443f\") "
Sep 19 08:34:43.291472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.291441    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data" (OuterVolumeSpecName: "data") pod "0c664b71-d252-4d8a-9020-b5d4482d443f" (UID: "0c664b71-d252-4d8a-9020-b5d4482d443f"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:43.292507 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.292469    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl" (OuterVolumeSpecName: "kube-api-access-pbjrl") pod "0c664b71-d252-4d8a-9020-b5d4482d443f" (UID: "0c664b71-d252-4d8a-9020-b5d4482d443f"). InnerVolumeSpecName "kube-api-access-pbjrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:43.385104 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.385081    2584 generic.go:356] "Generic (PLEG): container finished" podID="0c664b71-d252-4d8a-9020-b5d4482d443f" containerID="28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49" exitCode=0
Sep 19 08:34:43.385202 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.385121    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"
Sep 19 08:34:43.385202 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.385159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" event={"ID":"0c664b71-d252-4d8a-9020-b5d4482d443f","Type":"ContainerDied","Data":"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"}
Sep 19 08:34:43.385275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.385207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw" event={"ID":"0c664b71-d252-4d8a-9020-b5d4482d443f","Type":"ContainerDied","Data":"1d4ac7a160e00c618f064aa512f8fe2c7da351bc4f66df3da9f962ca8d63ea56"}
Sep 19 08:34:43.385275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.385230    2584 scope.go:117] "RemoveContainer" containerID="28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"
Sep 19 08:34:43.391019 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.390993    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjrl\" (UniqueName: \"kubernetes.io/projected/0c664b71-d252-4d8a-9020-b5d4482d443f-kube-api-access-pbjrl\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:34:43.391019 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.391017    2584 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0c664b71-d252-4d8a-9020-b5d4482d443f-data\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:34:43.395025 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.394975    2584 scope.go:117] "RemoveContainer" containerID="28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"
Sep 19 08:34:43.395314 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:34:43.395297    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49\": container with ID starting with 28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49 not found: ID does not exist" containerID="28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"
Sep 19 08:34:43.395380 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.395322    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49"} err="failed to get container status \"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49\": rpc error: code = NotFound desc = could not find container \"28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49\": container with ID starting with 28386dfcb8e35f2fd3d804047d94c1701aa4a7e5b976120b6305caa1b39cdc49 not found: ID does not exist"
Sep 19 08:34:43.404926 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.404905    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:43.408379 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.408362    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mmwtw"]
Sep 19 08:34:43.436729 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.436711    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"]
Sep 19 08:34:43.437072 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.437059    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c664b71-d252-4d8a-9020-b5d4482d443f" containerName="seaweedfs-tls-custom"
Sep 19 08:34:43.437121 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.437073    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c664b71-d252-4d8a-9020-b5d4482d443f" containerName="seaweedfs-tls-custom"
Sep 19 08:34:43.437155 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.437138    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c664b71-d252-4d8a-9020-b5d4482d443f" containerName="seaweedfs-tls-custom"
Sep 19 08:34:43.441304 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.441291    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.443514 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.443495    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 19 08:34:43.443606 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.443517    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 19 08:34:43.447882 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.447862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"]
Sep 19 08:34:43.593431 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.593377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.593431 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.593410    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvd7\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-kube-api-access-bnvd7\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.593562 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.593477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6224730d-13e7-411b-bfb1-0167fef2fcfc-data\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.694721 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.694697    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6224730d-13e7-411b-bfb1-0167fef2fcfc-data\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.694826 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.694753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.694826 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.694790    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvd7\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-kube-api-access-bnvd7\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.695055 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.695035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6224730d-13e7-411b-bfb1-0167fef2fcfc-data\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.697076 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.697058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.702665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.702644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvd7\" (UniqueName: \"kubernetes.io/projected/6224730d-13e7-411b-bfb1-0167fef2fcfc-kube-api-access-bnvd7\") pod \"seaweedfs-tls-custom-8b44d767d-cjjfk\" (UID: \"6224730d-13e7-411b-bfb1-0167fef2fcfc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.750226 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.750205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"
Sep 19 08:34:43.906472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:43.906409    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c664b71-d252-4d8a-9020-b5d4482d443f" path="/var/lib/kubelet/pods/0c664b71-d252-4d8a-9020-b5d4482d443f/volumes"
Sep 19 08:34:44.016506 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:44.016419    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-cjjfk"]
Sep 19 08:34:44.018955 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:34:44.018929    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6224730d_13e7_411b_bfb1_0167fef2fcfc.slice/crio-9b9af77da89520e1000b3d603455611cf1e5c6a5768413523cbf02ab1c3b4a2d WatchSource:0}: Error finding container 9b9af77da89520e1000b3d603455611cf1e5c6a5768413523cbf02ab1c3b4a2d: Status 404 returned error can't find the container with id 9b9af77da89520e1000b3d603455611cf1e5c6a5768413523cbf02ab1c3b4a2d
Sep 19 08:34:44.392214 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:44.392183    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk" event={"ID":"6224730d-13e7-411b-bfb1-0167fef2fcfc","Type":"ContainerStarted","Data":"4ee33cd6839c81b25e5f629c02056587c9539c8d2d99dd6f6098ce34c564aa33"}
Sep 19 08:34:44.392538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:44.392230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk" event={"ID":"6224730d-13e7-411b-bfb1-0167fef2fcfc","Type":"ContainerStarted","Data":"9b9af77da89520e1000b3d603455611cf1e5c6a5768413523cbf02ab1c3b4a2d"}
Sep 19 08:34:44.408639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:44.408595    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-cjjfk" podStartSLOduration=1.174174017 podStartE2EDuration="1.408580261s" totalImagesPullingTime="234.406244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:44.40665413 +0000 UTC m=+852.491154932" watchObservedRunningTime="2026-09-19 08:34:44.408580261 +0000 UTC m=+852.493081062"
Sep 19 08:34:56.987428 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:56.987395    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-llzst"]
Sep 19 08:34:56.990790 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:56.990774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:56.993130 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:56.993110    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 19 08:34:56.993130 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:56.993122    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 19 08:34:56.998461 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:56.998437    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-llzst"]
Sep 19 08:34:57.096958 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.096932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.097298 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.096960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x9w\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-kube-api-access-s4x9w\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.097298 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.097072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db21908-fbd2-4a6e-8e08-2ed2e313a177-data\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.198406 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.198387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.198508 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.198412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x9w\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-kube-api-access-s4x9w\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.198584 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.198551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db21908-fbd2-4a6e-8e08-2ed2e313a177-data\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.198879 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.198862    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1db21908-fbd2-4a6e-8e08-2ed2e313a177-data\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.200673 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.200653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.206008 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.205977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x9w\" (UniqueName: \"kubernetes.io/projected/1db21908-fbd2-4a6e-8e08-2ed2e313a177-kube-api-access-s4x9w\") pod \"seaweedfs-tls-serving-67b7477c6d-llzst\" (UID: \"1db21908-fbd2-4a6e-8e08-2ed2e313a177\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.302189 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.302140    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst"
Sep 19 08:34:57.568450 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:57.568428    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-llzst"]
Sep 19 08:34:57.570447 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:34:57.570419    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db21908_fbd2_4a6e_8e08_2ed2e313a177.slice/crio-23ff98876ec621e28ec045c3ca4288e4d0c834329767f58b5de6c33193220ffb WatchSource:0}: Error finding container 23ff98876ec621e28ec045c3ca4288e4d0c834329767f58b5de6c33193220ffb: Status 404 returned error can't find the container with id 23ff98876ec621e28ec045c3ca4288e4d0c834329767f58b5de6c33193220ffb
Sep 19 08:34:58.460136 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:58.460105    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst" event={"ID":"1db21908-fbd2-4a6e-8e08-2ed2e313a177","Type":"ContainerStarted","Data":"48ae5efa29681e6c39c4e0e9d39c27355d14f59a7cef699574845e7b1a17f8db"}
Sep 19 08:34:58.460512 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:58.460158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst" event={"ID":"1db21908-fbd2-4a6e-8e08-2ed2e313a177","Type":"ContainerStarted","Data":"23ff98876ec621e28ec045c3ca4288e4d0c834329767f58b5de6c33193220ffb"}
Sep 19 08:34:58.475917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:34:58.475865    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-llzst" podStartSLOduration=2.224030199 podStartE2EDuration="2.475848192s" totalImagesPullingTime="251.817993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:58.475179027 +0000 UTC m=+866.559679829" watchObservedRunningTime="2026-09-19 08:34:58.475848192 +0000 UTC m=+866.560348993"
Sep 19 08:35:23.510177 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.510090    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:23.517053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.515388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.518526    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b5f088-dockercfg-ksx9g\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.518667    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b5f088-kube-rbac-proxy-sar-config\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.518761    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b5f088-predictor-serving-cert\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.518957    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b5f088\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.519392    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.519641    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:35:23.521326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.519868    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:35:23.527715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.527679    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:23.593935 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.593878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.594194 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.594021    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmwc\" (UniqueName: \"kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.594194 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.594098    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.594194 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.594146    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.594194 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.594180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695197 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695472 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:35:23.695325    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-serving-cert: secret "isvc-init-fail-b5f088-predictor-serving-cert" not found
Sep 19 08:35:23.695472 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:35:23.695392    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls podName:65e07cc7-59a5-4108-97a7-b4d8fc08999d nodeName:}" failed. No retries permitted until 2026-09-19 08:35:24.195373718 +0000 UTC m=+892.279874511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls") pod "isvc-init-fail-b5f088-predictor-69584c5d56-mn528" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d") : secret "isvc-init-fail-b5f088-predictor-serving-cert" not found
Sep 19 08:35:23.695472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695410    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmwc\" (UniqueName: \"kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695915 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695896    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.695975 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.695918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:23.703589 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:23.703572    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmwc\" (UniqueName: \"kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:24.198654 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:24.198625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:24.200904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:24.200882    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") pod \"isvc-init-fail-b5f088-predictor-69584c5d56-mn528\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") " pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:24.450600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:24.450545    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:24.725607 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:24.725582    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:24.727605 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:35:24.727573    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e07cc7_59a5_4108_97a7_b4d8fc08999d.slice/crio-018d9f40bbbb75e29d1527b6043ba652493e92db3457f7f42a4184af8f6270cc WatchSource:0}: Error finding container 018d9f40bbbb75e29d1527b6043ba652493e92db3457f7f42a4184af8f6270cc: Status 404 returned error can't find the container with id 018d9f40bbbb75e29d1527b6043ba652493e92db3457f7f42a4184af8f6270cc
Sep 19 08:35:25.588472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:25.588435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerStarted","Data":"018d9f40bbbb75e29d1527b6043ba652493e92db3457f7f42a4184af8f6270cc"}
Sep 19 08:35:28.606258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:28.606225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerStarted","Data":"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"}
Sep 19 08:35:30.623834 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:30.623807    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/0.log"
Sep 19 08:35:30.624263 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:30.623844    2584 generic.go:356] "Generic (PLEG): container finished" podID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerID="27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c" exitCode=1
Sep 19 08:35:30.624263 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:30.623924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerDied","Data":"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"}
Sep 19 08:35:31.630395 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:31.630365    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/0.log"
Sep 19 08:35:31.630847 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:31.630444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerStarted","Data":"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"}
Sep 19 08:35:33.156238 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.156206    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:33.156564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.156448    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer" containerID="cri-o://aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593" gracePeriod=30
Sep 19 08:35:33.283672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.283651    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/1.log"
Sep 19 08:35:33.284123 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.284108    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/0.log"
Sep 19 08:35:33.286614 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.286599    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:33.374632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374612    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config\") pod \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") "
Sep 19 08:35:33.374745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374641    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert\") pod \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") "
Sep 19 08:35:33.374745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374665    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") pod \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") "
Sep 19 08:35:33.374745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374733    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdmwc\" (UniqueName: \"kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc\") pod \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") "
Sep 19 08:35:33.374861 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374748    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location\") pod \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\" (UID: \"65e07cc7-59a5-4108-97a7-b4d8fc08999d\") "
Sep 19 08:35:33.375015 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.374965    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65e07cc7-59a5-4108-97a7-b4d8fc08999d" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:33.375094 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.375036    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65e07cc7-59a5-4108-97a7-b4d8fc08999d" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:33.375164 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.375134    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65e07cc7-59a5-4108-97a7-b4d8fc08999d" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:35:33.376741 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.376718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65e07cc7-59a5-4108-97a7-b4d8fc08999d" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:35:33.376870 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.376847    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc" (OuterVolumeSpecName: "kube-api-access-zdmwc") pod "65e07cc7-59a5-4108-97a7-b4d8fc08999d" (UID: "65e07cc7-59a5-4108-97a7-b4d8fc08999d"). InnerVolumeSpecName "kube-api-access-zdmwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:35:33.476166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.476142    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmwc\" (UniqueName: \"kubernetes.io/projected/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-api-access-zdmwc\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:35:33.476166 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.476164    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:35:33.476285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.476173    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:35:33.476285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.476182    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65e07cc7-59a5-4108-97a7-b4d8fc08999d-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:35:33.476285 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.476190    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65e07cc7-59a5-4108-97a7-b4d8fc08999d-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:35:33.641734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.641686    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/1.log"
Sep 19 08:35:33.642114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642101    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b5f088-predictor-69584c5d56-mn528_65e07cc7-59a5-4108-97a7-b4d8fc08999d/storage-initializer/0.log"
Sep 19 08:35:33.642213 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642132    2584 generic.go:356] "Generic (PLEG): container finished" podID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerID="aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593" exitCode=1
Sep 19 08:35:33.642213 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerDied","Data":"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"}
Sep 19 08:35:33.642213 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642185    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"
Sep 19 08:35:33.642390 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528" event={"ID":"65e07cc7-59a5-4108-97a7-b4d8fc08999d","Type":"ContainerDied","Data":"018d9f40bbbb75e29d1527b6043ba652493e92db3457f7f42a4184af8f6270cc"}
Sep 19 08:35:33.642390 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.642285    2584 scope.go:117] "RemoveContainer" containerID="aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"
Sep 19 08:35:33.651553 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.651538    2584 scope.go:117] "RemoveContainer" containerID="27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"
Sep 19 08:35:33.659193 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.659173    2584 scope.go:117] "RemoveContainer" containerID="aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"
Sep 19 08:35:33.659454 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:35:33.659431    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593\": container with ID starting with aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593 not found: ID does not exist" containerID="aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"
Sep 19 08:35:33.659537 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.659455    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593"} err="failed to get container status \"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593\": rpc error: code = NotFound desc = could not find container \"aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593\": container with ID starting with aec81656b1e34c1f5ab43cef5bdaa61cd2869784380fcdf015be480c305f1593 not found: ID does not exist"
Sep 19 08:35:33.659537 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.659471    2584 scope.go:117] "RemoveContainer" containerID="27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"
Sep 19 08:35:33.659687 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:35:33.659669    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c\": container with ID starting with 27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c not found: ID does not exist" containerID="27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"
Sep 19 08:35:33.659731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.659691    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c"} err="failed to get container status \"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c\": rpc error: code = NotFound desc = could not find container \"27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c\": container with ID starting with 27c8cdd2731068e058ac0e9eef79abde057117d324ba2f6fdb671f1c599bb00c not found: ID does not exist"
Sep 19 08:35:33.678822 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.678797    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:33.683046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.683016    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b5f088-predictor-69584c5d56-mn528"]
Sep 19 08:35:33.908631 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.908553    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" path="/var/lib/kubelet/pods/65e07cc7-59a5-4108-97a7-b4d8fc08999d/volumes"
Sep 19 08:35:33.925913 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.925888    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:35:33.927722 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.927694    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:35:33.931253 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.931231    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:35:33.933039 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:33.933023    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:35:38.412696 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.412661    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:35:38.413216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413201    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.413258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413219    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.413258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413240    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.413258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413248    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.413348 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413325    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.413499 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.413488    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e07cc7-59a5-4108-97a7-b4d8fc08999d" containerName="storage-initializer"
Sep 19 08:35:38.426119 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.426087    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:35:38.426254 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.426229    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.428898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.428875    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:35:38.429043 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.428878    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:35:38.429043 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.428920    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:35:38.429247 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.429218    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 19 08:35:38.429375 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.429358    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:35:38.429375 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.429367    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:35:38.430211 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.430189    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 19 08:35:38.521523 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.521500    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmq6r\" (UniqueName: \"kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.521635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.521541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.521635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.521559    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.521756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.521666    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.521756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.521694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.622807 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.622772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.622807 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.622805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623038 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.622942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmq6r\" (UniqueName: \"kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623038 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.623030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623148 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.623049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623200 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.623183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623553 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.623523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.623711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.623652    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.625455 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.625438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.630895 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.630872    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmq6r\" (UniqueName: \"kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rtppk\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:38.738636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:38.738618    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:35:39.009281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:39.009258    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:35:39.011560 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:35:39.011536    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0232bc9b_1bc1_43f5_956d_f6d6dd645dbf.slice/crio-ac0f9fb74142441cd0f2a9805f73a2dd40f6c3d7b4ecfceebea4219ee977238e WatchSource:0}: Error finding container ac0f9fb74142441cd0f2a9805f73a2dd40f6c3d7b4ecfceebea4219ee977238e: Status 404 returned error can't find the container with id ac0f9fb74142441cd0f2a9805f73a2dd40f6c3d7b4ecfceebea4219ee977238e
Sep 19 08:35:39.671997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:39.671947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerStarted","Data":"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"}
Sep 19 08:35:39.672371 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:39.672026    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerStarted","Data":"ac0f9fb74142441cd0f2a9805f73a2dd40f6c3d7b4ecfceebea4219ee977238e"}
Sep 19 08:35:43.691830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:43.691791    2584 generic.go:356] "Generic (PLEG): container finished" podID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerID="18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172" exitCode=0
Sep 19 08:35:43.692258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:35:43.691862    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerDied","Data":"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"}
Sep 19 08:36:04.831611 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:04.831566    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerStarted","Data":"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"}
Sep 19 08:36:06.846312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:06.846276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerStarted","Data":"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"}
Sep 19 08:36:06.846728 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:06.846405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:36:06.868401 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:06.868351    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podStartSLOduration=2.616729886 podStartE2EDuration="28.868334637s" totalImagesPullingTime="22.578834908s" totalInitContainerRuntime="3.672769843s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:36:06.86524198 +0000 UTC m=+934.949742781" watchObservedRunningTime="2026-09-19 08:36:06.868334637 +0000 UTC m=+934.952835440"
Sep 19 08:36:07.851286 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:07.851257    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:36:07.852280 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:07.852259    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:08.856365 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:08.856323    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:13.861227 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:13.861165    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:36:14.071879 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:13.861698    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:23.861909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:23.861870    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:33.862088 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:33.862041    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:43.862525 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:43.862482    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:36:53.862667 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:36:53.862576    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:37:03.862187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:03.862143    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:37:13.862089 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:13.862013    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 19 08:37:23.862668 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:23.862635    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:37:28.896249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:28.896216    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:37:28.896689 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:28.896592    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container" containerID="cri-o://0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1" gracePeriod=30
Sep 19 08:37:28.896689 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:28.896631    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kube-rbac-proxy" containerID="cri-o://54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc" gracePeriod=30
Sep 19 08:37:29.237085 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:29.237057    2584 generic.go:356] "Generic (PLEG): container finished" podID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerID="54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc" exitCode=2
Sep 19 08:37:29.237216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:29.237131    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerDied","Data":"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"}
Sep 19 08:37:33.231604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.231583    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:37:33.258573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.258547    2584 generic.go:356] "Generic (PLEG): container finished" podID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerID="0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1" exitCode=0
Sep 19 08:37:33.258688 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.258611    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"
Sep 19 08:37:33.258688 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.258633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerDied","Data":"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"}
Sep 19 08:37:33.258765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.258700    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk" event={"ID":"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf","Type":"ContainerDied","Data":"ac0f9fb74142441cd0f2a9805f73a2dd40f6c3d7b4ecfceebea4219ee977238e"}
Sep 19 08:37:33.258765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.258739    2584 scope.go:117] "RemoveContainer" containerID="54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"
Sep 19 08:37:33.266807 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.266786    2584 scope.go:117] "RemoveContainer" containerID="0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"
Sep 19 08:37:33.276025 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.276009    2584 scope.go:117] "RemoveContainer" containerID="18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"
Sep 19 08:37:33.283349 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.283333    2584 scope.go:117] "RemoveContainer" containerID="54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"
Sep 19 08:37:33.283614 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:37:33.283596    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc\": container with ID starting with 54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc not found: ID does not exist" containerID="54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"
Sep 19 08:37:33.283672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.283624    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc"} err="failed to get container status \"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc\": rpc error: code = NotFound desc = could not find container \"54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc\": container with ID starting with 54e3be7d22dc3cb5f14b3aa8e61a75e7e4299fb4326529b84d99bc887c65e8dc not found: ID does not exist"
Sep 19 08:37:33.283672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.283640    2584 scope.go:117] "RemoveContainer" containerID="0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"
Sep 19 08:37:33.283855 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:37:33.283839    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1\": container with ID starting with 0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1 not found: ID does not exist" containerID="0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"
Sep 19 08:37:33.283896 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.283860    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1"} err="failed to get container status \"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1\": rpc error: code = NotFound desc = could not find container \"0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1\": container with ID starting with 0babb3b7b00e7d45d0c3889a009fed60cc8598730e5f985c8dc4e2218103dca1 not found: ID does not exist"
Sep 19 08:37:33.283896 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.283872    2584 scope.go:117] "RemoveContainer" containerID="18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"
Sep 19 08:37:33.284175 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:37:33.284157    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172\": container with ID starting with 18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172 not found: ID does not exist" containerID="18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"
Sep 19 08:37:33.284234 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.284182    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172"} err="failed to get container status \"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172\": rpc error: code = NotFound desc = could not find container \"18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172\": container with ID starting with 18fa370a2dcd22f52486e51d526beb39e41f4b9bd96899a553b7e0f437590172 not found: ID does not exist"
Sep 19 08:37:33.329256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location\") pod \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") "
Sep 19 08:37:33.329347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329281    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config\") pod \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") "
Sep 19 08:37:33.329347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329313    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert\") pod \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") "
Sep 19 08:37:33.329459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329360    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmq6r\" (UniqueName: \"kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r\") pod \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") "
Sep 19 08:37:33.329459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329380    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls\") pod \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\" (UID: \"0232bc9b-1bc1-43f5-956d-f6d6dd645dbf\") "
Sep 19 08:37:33.329604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" (UID: "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:33.329604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329594    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" (UID: "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:33.329685 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.329667    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" (UID: "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:33.331262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.331241    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" (UID: "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:33.331326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.331309    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r" (OuterVolumeSpecName: "kube-api-access-gmq6r") pod "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" (UID: "0232bc9b-1bc1-43f5-956d-f6d6dd645dbf"). InnerVolumeSpecName "kube-api-access-gmq6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:33.430848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.430801    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:37:33.430848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.430821    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmq6r\" (UniqueName: \"kubernetes.io/projected/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-api-access-gmq6r\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:37:33.430848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.430830    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:37:33.430848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.430840    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:37:33.430848 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.430848    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:37:33.580425 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.580402    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:37:33.584424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.584402    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rtppk"]
Sep 19 08:37:33.912539 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:33.912507    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" path="/var/lib/kubelet/pods/0232bc9b-1bc1-43f5-956d-f6d6dd645dbf/volumes"
Sep 19 08:37:34.140085 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140055    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:37:34.140508 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140491    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="storage-initializer"
Sep 19 08:37:34.140571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140510    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="storage-initializer"
Sep 19 08:37:34.140571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140528    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container"
Sep 19 08:37:34.140571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140534    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container"
Sep 19 08:37:34.140571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140546    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kube-rbac-proxy"
Sep 19 08:37:34.140571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140552    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kube-rbac-proxy"
Sep 19 08:37:34.140727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140605    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kserve-container"
Sep 19 08:37:34.140727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.140614    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0232bc9b-1bc1-43f5-956d-f6d6dd645dbf" containerName="kube-rbac-proxy"
Sep 19 08:37:34.143849 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.143828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.146268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.146226    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:37:34.146493 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.146474    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:37:34.146581 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.146530    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 19 08:37:34.146581 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.146534    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:37:34.146581 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.146558    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 19 08:37:34.147226 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.147202    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:37:34.147313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.147249    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:37:34.154505 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.154481    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:37:34.238387 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.238362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.238661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.238401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.238661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.238436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.238661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.238520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.238661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.238544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkr57\" (UniqueName: \"kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339556 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339556 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339556 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkr57\" (UniqueName: \"kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.339963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.339938    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.340151 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.340134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.340204 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.340185    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.341993 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.341963    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.347600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.347577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkr57\" (UniqueName: \"kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57\") pod \"isvc-predictive-xgboost-predictor-695f978978-kw8fl\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.457791 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.457767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:34.727788 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:34.727763    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:37:34.730742 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:37:34.730719    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0513ad3b_5f38_48cf_a903_d8c60f6575cb.slice/crio-c77c33afcb48f4b6e25007451837bd3de0cc2412b8023e8d737b49d058f52165 WatchSource:0}: Error finding container c77c33afcb48f4b6e25007451837bd3de0cc2412b8023e8d737b49d058f52165: Status 404 returned error can't find the container with id c77c33afcb48f4b6e25007451837bd3de0cc2412b8023e8d737b49d058f52165
Sep 19 08:37:35.271129 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:35.271096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerStarted","Data":"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"}
Sep 19 08:37:35.271571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:35.271160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerStarted","Data":"c77c33afcb48f4b6e25007451837bd3de0cc2412b8023e8d737b49d058f52165"}
Sep 19 08:37:39.291531 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:39.291497    2584 generic.go:356] "Generic (PLEG): container finished" podID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerID="454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c" exitCode=0
Sep 19 08:37:39.291901 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:39.291571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerDied","Data":"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"}
Sep 19 08:37:40.301917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:40.301880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerStarted","Data":"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"}
Sep 19 08:37:40.302321 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:40.301955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerStarted","Data":"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"}
Sep 19 08:37:40.302321 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:40.302200    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:40.322345 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:40.322303    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podStartSLOduration=2.659211475 podStartE2EDuration="6.32228912s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663077645s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:37:40.321202073 +0000 UTC m=+1028.405702876" watchObservedRunningTime="2026-09-19 08:37:40.32228912 +0000 UTC m=+1028.406789984"
Sep 19 08:37:41.307532 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:41.307505    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:41.308946 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:41.308918    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:37:42.312397 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:42.312359    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:37:47.317319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:47.317289    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:37:47.317784 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:47.317758    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:37:57.318107 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:37:57.318073    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:07.317738 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:07.317688    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:17.318230 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:17.318149    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:27.318413 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:27.318377    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:37.318664 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:37.318619    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:44.515180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.515139    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:38:44.519313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.519296    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.521828 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.521807    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 19 08:38:44.521918 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.521815    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 19 08:38:44.529185 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.529165    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:38:44.582553 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.582526    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.582654 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.582570    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.582695 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.582673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89s9\" (UniqueName: \"kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.683617 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.683591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89s9\" (UniqueName: \"kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.683715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.683650    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.683715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.683670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.684439 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.684417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.686116 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.686097    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.691426 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.691398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89s9\" (UniqueName: \"kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9\") pod \"message-dumper-predictor-6575bcfdfd-mnxf5\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:44.830099 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:44.830043    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:45.099663 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:45.099636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:38:45.101168 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:38:45.101144    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e54e816_a1bb_425c_ace2_755ec5000161.slice/crio-6d0a7be3d65a153d66a7f5f8eacab1d54bd27fabf1f8ee592f9c57dd0a4ce89f WatchSource:0}: Error finding container 6d0a7be3d65a153d66a7f5f8eacab1d54bd27fabf1f8ee592f9c57dd0a4ce89f: Status 404 returned error can't find the container with id 6d0a7be3d65a153d66a7f5f8eacab1d54bd27fabf1f8ee592f9c57dd0a4ce89f
Sep 19 08:38:45.103101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:45.103084    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:38:45.611148 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:45.611116    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerStarted","Data":"6d0a7be3d65a153d66a7f5f8eacab1d54bd27fabf1f8ee592f9c57dd0a4ce89f"}
Sep 19 08:38:46.617951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:46.617875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerStarted","Data":"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"}
Sep 19 08:38:46.617951 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:46.617932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerStarted","Data":"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"}
Sep 19 08:38:46.618362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:46.618034    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:46.635881 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:46.635837    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" podStartSLOduration=1.377557209 podStartE2EDuration="2.635824516s" totalImagesPullingTime="1.258267307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:38:46.633419026 +0000 UTC m=+1094.717919843" watchObservedRunningTime="2026-09-19 08:38:46.635824516 +0000 UTC m=+1094.720325318"
Sep 19 08:38:47.318668 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:47.318631    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:38:47.622867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:47.622775    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:47.624688 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:47.624666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:54.635754 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:54.635724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:38:57.318880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:38:57.318846    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:39:04.618607 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:04.618575    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:39:04.619066 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:04.618808    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" containerID="cri-o://d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2" gracePeriod=30
Sep 19 08:39:04.619066 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:04.618876    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kube-rbac-proxy" containerID="cri-o://4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde" gracePeriod=30
Sep 19 08:39:05.714337 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:05.714306    2584 generic.go:356] "Generic (PLEG): container finished" podID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerID="4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde" exitCode=2
Sep 19 08:39:05.714693 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:05.714379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerDied","Data":"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"}
Sep 19 08:39:07.313446 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:07.313402    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 19 08:39:07.317711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:07.317683    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 19 08:39:08.958543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:08.958519    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:39:09.080720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080661    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location\") pod \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") "
Sep 19 08:39:09.080720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080690    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config\") pod \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") "
Sep 19 08:39:09.080910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080752    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls\") pod \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") "
Sep 19 08:39:09.080910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080774    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert\") pod \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") "
Sep 19 08:39:09.080910 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080792    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkr57\" (UniqueName: \"kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57\") pod \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\" (UID: \"0513ad3b-5f38-48cf-a903-d8c60f6575cb\") "
Sep 19 08:39:09.081096 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.080963    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0513ad3b-5f38-48cf-a903-d8c60f6575cb" (UID: "0513ad3b-5f38-48cf-a903-d8c60f6575cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:39:09.081160 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.081127    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0513ad3b-5f38-48cf-a903-d8c60f6575cb" (UID: "0513ad3b-5f38-48cf-a903-d8c60f6575cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:09.081160 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.081141    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0513ad3b-5f38-48cf-a903-d8c60f6575cb" (UID: "0513ad3b-5f38-48cf-a903-d8c60f6575cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:09.082877 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.082852    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0513ad3b-5f38-48cf-a903-d8c60f6575cb" (UID: "0513ad3b-5f38-48cf-a903-d8c60f6575cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:09.082977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.082853    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57" (OuterVolumeSpecName: "kube-api-access-kkr57") pod "0513ad3b-5f38-48cf-a903-d8c60f6575cb" (UID: "0513ad3b-5f38-48cf-a903-d8c60f6575cb"). InnerVolumeSpecName "kube-api-access-kkr57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:09.181483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.181456    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:39:09.181483 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.181480    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkr57\" (UniqueName: \"kubernetes.io/projected/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-api-access-kkr57\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:39:09.181629 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.181493    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:39:09.181629 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.181508    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0513ad3b-5f38-48cf-a903-d8c60f6575cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:39:09.181629 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.181521    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0513ad3b-5f38-48cf-a903-d8c60f6575cb-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:39:09.738093 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.738062    2584 generic.go:356] "Generic (PLEG): container finished" podID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerID="d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2" exitCode=0
Sep 19 08:39:09.738269 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.738134    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"
Sep 19 08:39:09.738269 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.738148    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerDied","Data":"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"}
Sep 19 08:39:09.738269 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.738225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl" event={"ID":"0513ad3b-5f38-48cf-a903-d8c60f6575cb","Type":"ContainerDied","Data":"c77c33afcb48f4b6e25007451837bd3de0cc2412b8023e8d737b49d058f52165"}
Sep 19 08:39:09.738269 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.738263    2584 scope.go:117] "RemoveContainer" containerID="4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"
Sep 19 08:39:09.748797 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.748710    2584 scope.go:117] "RemoveContainer" containerID="d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"
Sep 19 08:39:09.757253 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.757235    2584 scope.go:117] "RemoveContainer" containerID="454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"
Sep 19 08:39:09.763576 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.763550    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:39:09.765841 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.765822    2584 scope.go:117] "RemoveContainer" containerID="4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"
Sep 19 08:39:09.766121 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:39:09.766102    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde\": container with ID starting with 4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde not found: ID does not exist" containerID="4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"
Sep 19 08:39:09.766181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.766134    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde"} err="failed to get container status \"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde\": rpc error: code = NotFound desc = could not find container \"4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde\": container with ID starting with 4d1938ad8dc4bd9e7956ec465631cf3fde8714f888323d50b6792b26362aadde not found: ID does not exist"
Sep 19 08:39:09.766181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.766151    2584 scope.go:117] "RemoveContainer" containerID="d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"
Sep 19 08:39:09.766397 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:39:09.766380    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2\": container with ID starting with d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2 not found: ID does not exist" containerID="d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"
Sep 19 08:39:09.766466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.766405    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2"} err="failed to get container status \"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2\": rpc error: code = NotFound desc = could not find container \"d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2\": container with ID starting with d4e29af871246a7f58bb7e77aa0e692371e999989b1a77702d380d71731d50c2 not found: ID does not exist"
Sep 19 08:39:09.766466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.766426    2584 scope.go:117] "RemoveContainer" containerID="454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"
Sep 19 08:39:09.766719 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:39:09.766683    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c\": container with ID starting with 454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c not found: ID does not exist" containerID="454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"
Sep 19 08:39:09.766790 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.766727    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c"} err="failed to get container status \"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c\": rpc error: code = NotFound desc = could not find container \"454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c\": container with ID starting with 454cca9e4c8d59955f1d4afc212fce8656f2fea5cc74e3894694e2cdb048f24c not found: ID does not exist"
Sep 19 08:39:09.769517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.769497    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kw8fl"]
Sep 19 08:39:09.775646 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.775625    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:39:09.776192 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776177    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kube-rbac-proxy"
Sep 19 08:39:09.776239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776195    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kube-rbac-proxy"
Sep 19 08:39:09.776239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776211    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container"
Sep 19 08:39:09.776239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776219    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container"
Sep 19 08:39:09.776239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776236    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="storage-initializer"
Sep 19 08:39:09.776361 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776245    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="storage-initializer"
Sep 19 08:39:09.776361 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776327    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kserve-container"
Sep 19 08:39:09.776361 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.776341    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" containerName="kube-rbac-proxy"
Sep 19 08:39:09.781538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.781519    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.787333 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.787296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 19 08:39:09.787494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.787459    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:39:09.787638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.787615    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:39:09.787868 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.787848    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 19 08:39:09.789804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.789778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:39:09.888199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.888172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.888295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.888210    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.888295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.888243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.888367 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.888315    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.888367 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.888348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxw9\" (UniqueName: \"kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.907113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.907089    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0513ad3b-5f38-48cf-a903-d8c60f6575cb" path="/var/lib/kubelet/pods/0513ad3b-5f38-48cf-a903-d8c60f6575cb/volumes"
Sep 19 08:39:09.989257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989257 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxw9\" (UniqueName: \"kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989622 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989622 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989622 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989622 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:39:09.989502    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 19 08:39:09.989622 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:39:09.989561    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls podName:2c2e63d6-d44a-4f97-850e-b28b8912844b nodeName:}" failed. No retries permitted until 2026-09-19 08:39:10.489541195 +0000 UTC m=+1118.574041996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 19 08:39:09.989852 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.989946 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.989924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.990045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.990029    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:09.998125 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:09.998104    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxw9\" (UniqueName: \"kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:10.493016 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:10.492978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:10.495323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:10.495302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:10.692438 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:10.692400    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:10.964034 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:10.964008    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:39:10.966435 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:39:10.966407    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e63d6_d44a_4f97_850e_b28b8912844b.slice/crio-6e4cbc397a9d2b11ac09202dc2aaf2126d5b3de22a2c85a6033cf1f72aaab6c9 WatchSource:0}: Error finding container 6e4cbc397a9d2b11ac09202dc2aaf2126d5b3de22a2c85a6033cf1f72aaab6c9: Status 404 returned error can't find the container with id 6e4cbc397a9d2b11ac09202dc2aaf2126d5b3de22a2c85a6033cf1f72aaab6c9
Sep 19 08:39:11.758319 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:11.758278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerStarted","Data":"8544ff547b3b27ff5bcc107b89400e858f9c30b2af541fc0458eabf13fc7755d"}
Sep 19 08:39:11.758695 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:11.758345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerStarted","Data":"6e4cbc397a9d2b11ac09202dc2aaf2126d5b3de22a2c85a6033cf1f72aaab6c9"}
Sep 19 08:39:14.774634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:14.774598    2584 generic.go:356] "Generic (PLEG): container finished" podID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerID="8544ff547b3b27ff5bcc107b89400e858f9c30b2af541fc0458eabf13fc7755d" exitCode=0
Sep 19 08:39:14.774961 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:14.774664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerDied","Data":"8544ff547b3b27ff5bcc107b89400e858f9c30b2af541fc0458eabf13fc7755d"}
Sep 19 08:39:15.781756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:15.781722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerStarted","Data":"99ef36f67b28fc01877da8e1cf23c38f48f8bcd0a36758f96a0982d5b7bb6b4f"}
Sep 19 08:39:15.782138 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:15.781785    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerStarted","Data":"ee24705d90386a319c820141427f1dd590bcafd3eb2131133e237ae0c093225d"}
Sep 19 08:39:15.782138 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:15.782005    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:15.804480 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:15.804432    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podStartSLOduration=3.107970162 podStartE2EDuration="6.804418471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.696448309s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:39:15.802019608 +0000 UTC m=+1123.886520410" watchObservedRunningTime="2026-09-19 08:39:15.804418471 +0000 UTC m=+1123.888919272"
Sep 19 08:39:16.786794 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:16.786757    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:16.788040 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:16.788013    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:39:17.791911 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:17.791874    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:39:22.796620 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:22.796589    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:39:22.797299 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:22.797268    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:39:32.798187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:32.798145    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:39:42.797567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:42.797522    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:39:52.797932 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:39:52.797838    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:40:02.797928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:02.797886    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:40:12.798147 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:12.798101    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:40:22.797329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:22.797290    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 19 08:40:28.902145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:28.902108    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:40:30.484950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:30.484917    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:40:30.485321 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:30.485276    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container" containerID="cri-o://ee24705d90386a319c820141427f1dd590bcafd3eb2131133e237ae0c093225d" gracePeriod=30
Sep 19 08:40:30.485398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:30.485360    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kube-rbac-proxy" containerID="cri-o://99ef36f67b28fc01877da8e1cf23c38f48f8bcd0a36758f96a0982d5b7bb6b4f" gracePeriod=30
Sep 19 08:40:31.144407 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:31.144374    2584 generic.go:356] "Generic (PLEG): container finished" podID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerID="99ef36f67b28fc01877da8e1cf23c38f48f8bcd0a36758f96a0982d5b7bb6b4f" exitCode=2
Sep 19 08:40:31.144574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:31.144418    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerDied","Data":"99ef36f67b28fc01877da8e1cf23c38f48f8bcd0a36758f96a0982d5b7bb6b4f"}
Sep 19 08:40:32.792429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:32.792388    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 19 08:40:33.959004 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:33.958962    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:40:33.961839 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:33.961815    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:40:33.964443 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:33.964421    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:40:33.967814 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:33.967794    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:40:35.166996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.166953    2584 generic.go:356] "Generic (PLEG): container finished" podID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerID="ee24705d90386a319c820141427f1dd590bcafd3eb2131133e237ae0c093225d" exitCode=0
Sep 19 08:40:35.167396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.167019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerDied","Data":"ee24705d90386a319c820141427f1dd590bcafd3eb2131133e237ae0c093225d"}
Sep 19 08:40:35.224665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.224646    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:40:35.358154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358128    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config\") pod \"2c2e63d6-d44a-4f97-850e-b28b8912844b\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") "
Sep 19 08:40:35.358154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358159    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location\") pod \"2c2e63d6-d44a-4f97-850e-b28b8912844b\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") "
Sep 19 08:40:35.358320 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358198    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert\") pod \"2c2e63d6-d44a-4f97-850e-b28b8912844b\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") "
Sep 19 08:40:35.358320 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358232    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmxw9\" (UniqueName: \"kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9\") pod \"2c2e63d6-d44a-4f97-850e-b28b8912844b\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") "
Sep 19 08:40:35.358425 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358405    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") pod \"2c2e63d6-d44a-4f97-850e-b28b8912844b\" (UID: \"2c2e63d6-d44a-4f97-850e-b28b8912844b\") "
Sep 19 08:40:35.358497 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358475    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c2e63d6-d44a-4f97-850e-b28b8912844b" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:35.358550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358511    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c2e63d6-d44a-4f97-850e-b28b8912844b" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:40:35.358600 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c2e63d6-d44a-4f97-850e-b28b8912844b" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:40:35.358753 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358739    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:35.358791 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358756    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:35.358791 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.358766    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2e63d6-d44a-4f97-850e-b28b8912844b-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:35.360277 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.360256    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9" (OuterVolumeSpecName: "kube-api-access-tmxw9") pod "2c2e63d6-d44a-4f97-850e-b28b8912844b" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b"). InnerVolumeSpecName "kube-api-access-tmxw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:35.360370 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.360331    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c2e63d6-d44a-4f97-850e-b28b8912844b" (UID: "2c2e63d6-d44a-4f97-850e-b28b8912844b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:35.459432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.459383    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxw9\" (UniqueName: \"kubernetes.io/projected/2c2e63d6-d44a-4f97-850e-b28b8912844b-kube-api-access-tmxw9\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:35.459432 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.459403    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e63d6-d44a-4f97-850e-b28b8912844b-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:35.678582 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678551    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:40:35.678925 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678913    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kube-rbac-proxy"
Sep 19 08:40:35.678965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678926    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kube-rbac-proxy"
Sep 19 08:40:35.678965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678937    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container"
Sep 19 08:40:35.678965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678942    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container"
Sep 19 08:40:35.678965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678951    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="storage-initializer"
Sep 19 08:40:35.678965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.678957    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="storage-initializer"
Sep 19 08:40:35.679134 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.679021    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kube-rbac-proxy"
Sep 19 08:40:35.679134 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.679030    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" containerName="kserve-container"
Sep 19 08:40:35.682379 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.682364    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.684612 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.684585    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 19 08:40:35.684727 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.684617    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 19 08:40:35.695170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.695129    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:40:35.762885 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.762866    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.763036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.762910    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzd94\" (UniqueName: \"kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.763036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.762936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.763036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.763025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.763220 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.763093    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863454 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863428    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzd94\" (UniqueName: \"kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863570 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863570 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863570 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863570 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.863773 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:40:35.863654    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 19 08:40:35.863773 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:40:35.863716    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls podName:f38852d2-f5d6-4e5b-af5c-ed33151e5cb3 nodeName:}" failed. No retries permitted until 2026-09-19 08:40:36.363697054 +0000 UTC m=+1204.448197860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 19 08:40:35.863968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.863951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.864104 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.864087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.864165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.864142    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:35.871681 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:35.871663    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzd94\" (UniqueName: \"kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:36.173638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.173562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx" event={"ID":"2c2e63d6-d44a-4f97-850e-b28b8912844b","Type":"ContainerDied","Data":"6e4cbc397a9d2b11ac09202dc2aaf2126d5b3de22a2c85a6033cf1f72aaab6c9"}
Sep 19 08:40:36.173638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.173592    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"
Sep 19 08:40:36.173638 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.173632    2584 scope.go:117] "RemoveContainer" containerID="99ef36f67b28fc01877da8e1cf23c38f48f8bcd0a36758f96a0982d5b7bb6b4f"
Sep 19 08:40:36.181880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.181862    2584 scope.go:117] "RemoveContainer" containerID="ee24705d90386a319c820141427f1dd590bcafd3eb2131133e237ae0c093225d"
Sep 19 08:40:36.189225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.189204    2584 scope.go:117] "RemoveContainer" containerID="8544ff547b3b27ff5bcc107b89400e858f9c30b2af541fc0458eabf13fc7755d"
Sep 19 08:40:36.190865 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.190845    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:40:36.194302 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.194284    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4rcxx"]
Sep 19 08:40:36.368803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.368768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:36.371103 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.371086    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:36.593054 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.593029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:36.866819 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:36.866798    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:40:36.868995 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:40:36.868953    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38852d2_f5d6_4e5b_af5c_ed33151e5cb3.slice/crio-3d4ebb388665e8538717c8be7112a5635273ceb3cd79a8b8f7774866c8bee836 WatchSource:0}: Error finding container 3d4ebb388665e8538717c8be7112a5635273ceb3cd79a8b8f7774866c8bee836: Status 404 returned error can't find the container with id 3d4ebb388665e8538717c8be7112a5635273ceb3cd79a8b8f7774866c8bee836
Sep 19 08:40:37.180832 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:37.180735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerStarted","Data":"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"}
Sep 19 08:40:37.180832 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:37.180793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerStarted","Data":"3d4ebb388665e8538717c8be7112a5635273ceb3cd79a8b8f7774866c8bee836"}
Sep 19 08:40:37.908250 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:37.908217    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2e63d6-d44a-4f97-850e-b28b8912844b" path="/var/lib/kubelet/pods/2c2e63d6-d44a-4f97-850e-b28b8912844b/volumes"
Sep 19 08:40:40.030346 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.030310    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-mnxf5_3e54e816-a1bb-425c-ace2-755ec5000161/kserve-container/0.log"
Sep 19 08:40:40.281248 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.281172    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:40:40.281421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.281397    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kserve-container" containerID="cri-o://67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb" gracePeriod=30
Sep 19 08:40:40.281490 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.281412    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kube-rbac-proxy" containerID="cri-o://a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e" gracePeriod=30
Sep 19 08:40:40.401573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.401547    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:40:40.405249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.405232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.407767 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.407740    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 19 08:40:40.408063 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.408042    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 19 08:40:40.415813 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.415788    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:40:40.505272 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.505242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kvt\" (UniqueName: \"kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.505426 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.505313    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.505426 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.505408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.505549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.505431    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.505549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.505457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.533487 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.533435    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:40:40.606830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.606795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config\") pod \"3e54e816-a1bb-425c-ace2-755ec5000161\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") "
Sep 19 08:40:40.606965 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.606910    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls\") pod \"3e54e816-a1bb-425c-ace2-755ec5000161\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") "
Sep 19 08:40:40.607041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v89s9\" (UniqueName: \"kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9\") pod \"3e54e816-a1bb-425c-ace2-755ec5000161\" (UID: \"3e54e816-a1bb-425c-ace2-755ec5000161\") "
Sep 19 08:40:40.607180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607161    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.607275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e54e816-a1bb-425c-ace2-755ec5000161" (UID: "3e54e816-a1bb-425c-ace2-755ec5000161"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:40:40.607275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607254    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.607389 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.607389 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607382    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.607496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kvt\" (UniqueName: \"kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.607543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607526    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e54e816-a1bb-425c-ace2-755ec5000161-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:40.607667 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.607636    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.608216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.608168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.608309 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.608250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.609795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.609774    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e54e816-a1bb-425c-ace2-755ec5000161" (UID: "3e54e816-a1bb-425c-ace2-755ec5000161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:40.609795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.609779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9" (OuterVolumeSpecName: "kube-api-access-v89s9") pod "3e54e816-a1bb-425c-ace2-755ec5000161" (UID: "3e54e816-a1bb-425c-ace2-755ec5000161"). InnerVolumeSpecName "kube-api-access-v89s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:40.610044 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.610026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.615163 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.615141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kvt\" (UniqueName: \"kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2g7f8\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.708932 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.708903    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e54e816-a1bb-425c-ace2-755ec5000161-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:40.708932 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.708929    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89s9\" (UniqueName: \"kubernetes.io/projected/3e54e816-a1bb-425c-ace2-755ec5000161-kube-api-access-v89s9\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:40:40.716826 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.716803    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:40:40.987897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:40.987876    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:40:40.989356 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:40:40.989333    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb89998a_c4fa_4d12_bd6d_d44a4dde58c6.slice/crio-398607b78a87ae9c412347a0f1691a0792c25ba1a4ead42175cac3ca98992e76 WatchSource:0}: Error finding container 398607b78a87ae9c412347a0f1691a0792c25ba1a4ead42175cac3ca98992e76: Status 404 returned error can't find the container with id 398607b78a87ae9c412347a0f1691a0792c25ba1a4ead42175cac3ca98992e76
Sep 19 08:40:41.207472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207392    2584 generic.go:356] "Generic (PLEG): container finished" podID="3e54e816-a1bb-425c-ace2-755ec5000161" containerID="a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e" exitCode=2
Sep 19 08:40:41.207472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207417    2584 generic.go:356] "Generic (PLEG): container finished" podID="3e54e816-a1bb-425c-ace2-755ec5000161" containerID="67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb" exitCode=2
Sep 19 08:40:41.207472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207452    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"
Sep 19 08:40:41.207472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerDied","Data":"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"}
Sep 19 08:40:41.208026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerDied","Data":"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"}
Sep 19 08:40:41.208026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5" event={"ID":"3e54e816-a1bb-425c-ace2-755ec5000161","Type":"ContainerDied","Data":"6d0a7be3d65a153d66a7f5f8eacab1d54bd27fabf1f8ee592f9c57dd0a4ce89f"}
Sep 19 08:40:41.208026 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.207597    2584 scope.go:117] "RemoveContainer" containerID="a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"
Sep 19 08:40:41.209605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.209584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerStarted","Data":"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"}
Sep 19 08:40:41.209733 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.209636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerStarted","Data":"398607b78a87ae9c412347a0f1691a0792c25ba1a4ead42175cac3ca98992e76"}
Sep 19 08:40:41.211313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.211286    2584 generic.go:356] "Generic (PLEG): container finished" podID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerID="a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424" exitCode=0
Sep 19 08:40:41.211409 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.211349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerDied","Data":"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"}
Sep 19 08:40:41.218437 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.218421    2584 scope.go:117] "RemoveContainer" containerID="67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"
Sep 19 08:40:41.225892 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.225871    2584 scope.go:117] "RemoveContainer" containerID="a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"
Sep 19 08:40:41.226183 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:40:41.226165    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e\": container with ID starting with a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e not found: ID does not exist" containerID="a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"
Sep 19 08:40:41.226241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226190    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"} err="failed to get container status \"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e\": rpc error: code = NotFound desc = could not find container \"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e\": container with ID starting with a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e not found: ID does not exist"
Sep 19 08:40:41.226241 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226205    2584 scope.go:117] "RemoveContainer" containerID="67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"
Sep 19 08:40:41.226468 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:40:41.226452    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb\": container with ID starting with 67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb not found: ID does not exist" containerID="67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"
Sep 19 08:40:41.226513 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226470    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"} err="failed to get container status \"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb\": rpc error: code = NotFound desc = could not find container \"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb\": container with ID starting with 67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb not found: ID does not exist"
Sep 19 08:40:41.226513 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226481    2584 scope.go:117] "RemoveContainer" containerID="a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"
Sep 19 08:40:41.226734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226718    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e"} err="failed to get container status \"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e\": rpc error: code = NotFound desc = could not find container \"a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e\": container with ID starting with a86835779bc5de4706295b64fce1be53e6b21492ac3a0946e742fd766969239e not found: ID does not exist"
Sep 19 08:40:41.226822 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226733    2584 scope.go:117] "RemoveContainer" containerID="67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"
Sep 19 08:40:41.226971 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.226954    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb"} err="failed to get container status \"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb\": rpc error: code = NotFound desc = could not find container \"67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb\": container with ID starting with 67efe989187968ffa5ea1846f58b555376f8fdb1877897f7a7d15c651bed2cfb not found: ID does not exist"
Sep 19 08:40:41.257502 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.257480    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:40:41.261557 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.261532    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mnxf5"]
Sep 19 08:40:41.906958 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:41.906926    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" path="/var/lib/kubelet/pods/3e54e816-a1bb-425c-ace2-755ec5000161/volumes"
Sep 19 08:40:42.224135 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:42.224099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerStarted","Data":"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"}
Sep 19 08:40:42.224568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:42.224189    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerStarted","Data":"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"}
Sep 19 08:40:42.224568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:42.224393    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:42.224568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:42.224539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:40:42.243795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:42.243753    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podStartSLOduration=3.324983425 podStartE2EDuration="7.243737992s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918754567s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:40:42.243376654 +0000 UTC m=+1210.327877467" watchObservedRunningTime="2026-09-19 08:40:42.243737992 +0000 UTC m=+1210.328238792"
Sep 19 08:40:48.234662 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:40:48.234637    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:41:14.388992 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:14.388960    2584 generic.go:356] "Generic (PLEG): container finished" podID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerID="afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6" exitCode=0
Sep 19 08:41:14.389356 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:14.389041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerDied","Data":"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"}
Sep 19 08:41:18.236633 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:18.236546    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:41:28.236000 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:28.235947    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:41:38.235753 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:38.235716    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:41:48.235163 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:48.235118    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:41:58.235627 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:41:58.235584    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:08.883999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:08.883953    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:42:15.809207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:15.809132    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:42:15.809700 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:15.809426    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" containerID="cri-o://95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b" gracePeriod=30
Sep 19 08:42:15.809700 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:15.809593    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" containerID="cri-o://244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e" gracePeriod=30
Sep 19 08:42:18.230997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:18.230953    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:18.235338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:18.235309    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:23.230660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:23.230618    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:28.230689 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:28.230640    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:28.231160 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:28.230916    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:42:28.235860 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:28.235823    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:33.230905 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:33.230865    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:38.230824 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:38.230786    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:38.235255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:38.235219    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:38.235387 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:38.235343    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:42:43.230795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:43.230747    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:48.230716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:48.230628    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:48.236006 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:48.235960    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:53.230800 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:53.230754    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:58.230747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.230702    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 19 08:42:58.236064 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.236035    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 19 08:42:58.700311 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.700284    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:42:58.861997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.861917    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config\") pod \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") "
Sep 19 08:42:58.861997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.861961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzd94\" (UniqueName: \"kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94\") pod \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") "
Sep 19 08:42:58.861997 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.861978    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") pod \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") "
Sep 19 08:42:58.862215 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862123    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location\") pod \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") "
Sep 19 08:42:58.862259 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862229    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert\") pod \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\" (UID: \"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3\") "
Sep 19 08:42:58.862415 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862387    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:42:58.862538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862415    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:58.862607 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862584    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:42:58.862607 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862598    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:42:58.862731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.862645    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:42:58.863995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.863961    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:58.864096 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.864080    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94" (OuterVolumeSpecName: "kube-api-access-pzd94") pod "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" (UID: "f38852d2-f5d6-4e5b-af5c-ed33151e5cb3"). InnerVolumeSpecName "kube-api-access-pzd94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:58.962955 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.962929    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:42:58.962955 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.962953    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzd94\" (UniqueName: \"kubernetes.io/projected/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-kube-api-access-pzd94\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:42:58.963109 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.962967    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:42:58.981128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981103    2584 generic.go:356] "Generic (PLEG): container finished" podID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerID="244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e" exitCode=2
Sep 19 08:42:58.981128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981127    2584 generic.go:356] "Generic (PLEG): container finished" podID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerID="95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b" exitCode=0
Sep 19 08:42:58.981255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981170    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"
Sep 19 08:42:58.981255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerDied","Data":"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"}
Sep 19 08:42:58.981255 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerDied","Data":"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"}
Sep 19 08:42:58.981352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981271    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm" event={"ID":"f38852d2-f5d6-4e5b-af5c-ed33151e5cb3","Type":"ContainerDied","Data":"3d4ebb388665e8538717c8be7112a5635273ceb3cd79a8b8f7774866c8bee836"}
Sep 19 08:42:58.981352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.981295    2584 scope.go:117] "RemoveContainer" containerID="244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"
Sep 19 08:42:58.989672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.989658    2584 scope.go:117] "RemoveContainer" containerID="95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"
Sep 19 08:42:58.998736 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:58.998715    2584 scope.go:117] "RemoveContainer" containerID="a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"
Sep 19 08:42:59.003869 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.003850    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:42:59.007045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007023    2584 scope.go:117] "RemoveContainer" containerID="244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"
Sep 19 08:42:59.007336 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:42:59.007315    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e\": container with ID starting with 244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e not found: ID does not exist" containerID="244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"
Sep 19 08:42:59.007402 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007347    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"} err="failed to get container status \"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e\": rpc error: code = NotFound desc = could not find container \"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e\": container with ID starting with 244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e not found: ID does not exist"
Sep 19 08:42:59.007402 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007370    2584 scope.go:117] "RemoveContainer" containerID="95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"
Sep 19 08:42:59.007630 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:42:59.007612    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b\": container with ID starting with 95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b not found: ID does not exist" containerID="95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"
Sep 19 08:42:59.007669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007638    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"} err="failed to get container status \"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b\": rpc error: code = NotFound desc = could not find container \"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b\": container with ID starting with 95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b not found: ID does not exist"
Sep 19 08:42:59.007669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007655    2584 scope.go:117] "RemoveContainer" containerID="a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"
Sep 19 08:42:59.007745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007738    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-pd7zm"]
Sep 19 08:42:59.007880 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:42:59.007862    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424\": container with ID starting with a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424 not found: ID does not exist" containerID="a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"
Sep 19 08:42:59.007920 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007886    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"} err="failed to get container status \"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424\": rpc error: code = NotFound desc = could not find container \"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424\": container with ID starting with a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424 not found: ID does not exist"
Sep 19 08:42:59.007920 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.007900    2584 scope.go:117] "RemoveContainer" containerID="244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"
Sep 19 08:42:59.008172 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.008155    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e"} err="failed to get container status \"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e\": rpc error: code = NotFound desc = could not find container \"244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e\": container with ID starting with 244b0923323996e95176e8bc6e956f1d894ea5b4709a29c8a6371c387483f06e not found: ID does not exist"
Sep 19 08:42:59.008172 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.008172    2584 scope.go:117] "RemoveContainer" containerID="95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"
Sep 19 08:42:59.008398 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.008381    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b"} err="failed to get container status \"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b\": rpc error: code = NotFound desc = could not find container \"95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b\": container with ID starting with 95234f1a3e75b9f1d8daaee8ce727a5ffa9379587c152baf2262070c72836b0b not found: ID does not exist"
Sep 19 08:42:59.008449 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.008398    2584 scope.go:117] "RemoveContainer" containerID="a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"
Sep 19 08:42:59.008650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.008630    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424"} err="failed to get container status \"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424\": rpc error: code = NotFound desc = could not find container \"a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424\": container with ID starting with a262dfe9aba99794f42a506e08268bcd31bab5383f6e87dc43885abb2fae8424 not found: ID does not exist"
Sep 19 08:42:59.906578 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.906554    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" path="/var/lib/kubelet/pods/f38852d2-f5d6-4e5b-af5c-ed33151e5cb3/volumes"
Sep 19 08:42:59.996156 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.996123    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerStarted","Data":"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"}
Sep 19 08:42:59.996310 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.996213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerStarted","Data":"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"}
Sep 19 08:42:59.996412 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.996391    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:42:59.996556 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.996533    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:42:59.998055 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:42:59.998029    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 19 08:43:00.016512 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:00.016470    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podStartSLOduration=1.7366271869999999 podStartE2EDuration="2m20.016458696s" totalImagesPullingTime="1m45.181275784s" totalInitContainerRuntime="33.098555725s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:00.015603914 +0000 UTC m=+1348.100104715" watchObservedRunningTime="2026-09-19 08:43:00.016458696 +0000 UTC m=+1348.100959544"
Sep 19 08:43:01.001895 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:01.001855    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 19 08:43:06.006280 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:06.006252    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:43:06.006782 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:06.006757    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 19 08:43:16.007123 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:16.007096    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:43:26.665204 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:26.665167    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:43:26.665663 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:26.665428    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container" containerID="cri-o://61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803" gracePeriod=30
Sep 19 08:43:26.665663 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:26.665499    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kube-rbac-proxy" containerID="cri-o://1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d" gracePeriod=30
Sep 19 08:43:27.131494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:27.131457    2584 generic.go:356] "Generic (PLEG): container finished" podID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerID="1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d" exitCode=2
Sep 19 08:43:27.131663 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:27.131529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerDied","Data":"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"}
Sep 19 08:43:28.038744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.038724    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:43:28.099086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099030    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert\") pod \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") "
Sep 19 08:43:28.099086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099084    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54kvt\" (UniqueName: \"kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt\") pod \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") "
Sep 19 08:43:28.099264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config\") pod \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") "
Sep 19 08:43:28.099264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099144    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls\") pod \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") "
Sep 19 08:43:28.099264 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099173    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location\") pod \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\" (UID: \"db89998a-c4fa-4d12-bd6d-d44a4dde58c6\") "
Sep 19 08:43:28.099495 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099457    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db89998a-c4fa-4d12-bd6d-d44a4dde58c6" (UID: "db89998a-c4fa-4d12-bd6d-d44a4dde58c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:28.099495 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.099472    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db89998a-c4fa-4d12-bd6d-d44a4dde58c6" (UID: "db89998a-c4fa-4d12-bd6d-d44a4dde58c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:28.101168 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.101136    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt" (OuterVolumeSpecName: "kube-api-access-54kvt") pod "db89998a-c4fa-4d12-bd6d-d44a4dde58c6" (UID: "db89998a-c4fa-4d12-bd6d-d44a4dde58c6"). InnerVolumeSpecName "kube-api-access-54kvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:28.101248 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.101224    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db89998a-c4fa-4d12-bd6d-d44a4dde58c6" (UID: "db89998a-c4fa-4d12-bd6d-d44a4dde58c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:28.102372 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.102349    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db89998a-c4fa-4d12-bd6d-d44a4dde58c6" (UID: "db89998a-c4fa-4d12-bd6d-d44a4dde58c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:28.138235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.138209    2584 generic.go:356] "Generic (PLEG): container finished" podID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerID="61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803" exitCode=0
Sep 19 08:43:28.138322 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.138265    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"
Sep 19 08:43:28.138322 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.138290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerDied","Data":"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"}
Sep 19 08:43:28.138399 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.138348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8" event={"ID":"db89998a-c4fa-4d12-bd6d-d44a4dde58c6","Type":"ContainerDied","Data":"398607b78a87ae9c412347a0f1691a0792c25ba1a4ead42175cac3ca98992e76"}
Sep 19 08:43:28.138399 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.138373    2584 scope.go:117] "RemoveContainer" containerID="1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"
Sep 19 08:43:28.146626 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.146612    2584 scope.go:117] "RemoveContainer" containerID="61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"
Sep 19 08:43:28.153896 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.153878    2584 scope.go:117] "RemoveContainer" containerID="afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"
Sep 19 08:43:28.160802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.160779    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:43:28.161659 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.161643    2584 scope.go:117] "RemoveContainer" containerID="1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"
Sep 19 08:43:28.161942 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:43:28.161911    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d\": container with ID starting with 1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d not found: ID does not exist" containerID="1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"
Sep 19 08:43:28.162002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.161953    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d"} err="failed to get container status \"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d\": rpc error: code = NotFound desc = could not find container \"1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d\": container with ID starting with 1b3bbc01159c9d1c2f041bbb1390a86683d096db3ff29b11c40daadb2f26135d not found: ID does not exist"
Sep 19 08:43:28.162002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.161975    2584 scope.go:117] "RemoveContainer" containerID="61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"
Sep 19 08:43:28.162270 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:43:28.162251    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803\": container with ID starting with 61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803 not found: ID does not exist" containerID="61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"
Sep 19 08:43:28.162313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.162276    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803"} err="failed to get container status \"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803\": rpc error: code = NotFound desc = could not find container \"61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803\": container with ID starting with 61cab2e78d29daa999135094be158abf4d940d7a049e3063b2c0865cc5bf4803 not found: ID does not exist"
Sep 19 08:43:28.162313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.162291    2584 scope.go:117] "RemoveContainer" containerID="afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"
Sep 19 08:43:28.162519 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:43:28.162499    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6\": container with ID starting with afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6 not found: ID does not exist" containerID="afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"
Sep 19 08:43:28.162560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.162524    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6"} err="failed to get container status \"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6\": rpc error: code = NotFound desc = could not find container \"afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6\": container with ID starting with afcaa25a6ab28da796378043df6a2686d0d7732a61998e47e192e64dd6c294f6 not found: ID does not exist"
Sep 19 08:43:28.162560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.162510    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2g7f8"]
Sep 19 08:43:28.200111 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.200093    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:43:28.200111 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.200113    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54kvt\" (UniqueName: \"kubernetes.io/projected/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-api-access-54kvt\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:43:28.200221 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.200122    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:43:28.200221 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.200135    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:43:28.200221 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:28.200148    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db89998a-c4fa-4d12-bd6d-d44a4dde58c6-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:43:29.909438 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:29.909402    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" path="/var/lib/kubelet/pods/db89998a-c4fa-4d12-bd6d-d44a4dde58c6/volumes"
Sep 19 08:43:31.858703 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.858670    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:43:31.859071 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859056    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="storage-initializer"
Sep 19 08:43:31.859071 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859068    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="storage-initializer"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859078    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859084    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859092    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859098    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859104    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859109    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859120    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859124    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859130    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859134    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859141    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="storage-initializer"
Sep 19 08:43:31.859144 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859146    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="storage-initializer"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859157    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kserve-container"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859161    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kserve-container"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859215    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859223    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859231    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e54e816-a1bb-425c-ace2-755ec5000161" containerName="kserve-container"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859236    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kube-rbac-proxy"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859241    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="db89998a-c4fa-4d12-bd6d-d44a4dde58c6" containerName="kserve-container"
Sep 19 08:43:31.859744 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.859246    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38852d2-f5d6-4e5b-af5c-ed33151e5cb3" containerName="kserve-container"
Sep 19 08:43:31.896404 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.896381    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:43:31.896536 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.896481    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:31.900661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.900636    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:43:31.900802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.900714    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 19 08:43:31.901180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.901160    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 19 08:43:31.901180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.901169    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:43:31.901342 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.901276    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:43:31.901342 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.901277    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:43:31.901510 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:31.901494    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:43:32.027422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.027388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8c52\" (UniqueName: \"kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.027538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.027441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.027538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.027469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.027538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.027509    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.027660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.027612    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.128706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.128655    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c52\" (UniqueName: \"kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.128706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.128684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.128706 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.128700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.128851 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.128717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.128851 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.128782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.129154 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:43:32.128940    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 19 08:43:32.129154 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:43:32.129029    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls podName:e31c3831-d271-4db8-889e-bfe5f87224ba nodeName:}" failed. No retries permitted until 2026-09-19 08:43:32.629009279 +0000 UTC m=+1380.713510079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 19 08:43:32.129154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.129094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.129464 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.129444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.129503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.129445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.139157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.139134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c52\" (UniqueName: \"kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.631969 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.631939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.634318 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.634295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sgmzq\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:32.807519 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:32.807492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:33.096527 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:33.096503    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:43:33.098404 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:43:33.098375    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31c3831_d271_4db8_889e_bfe5f87224ba.slice/crio-0d837ec5ab4d957b09fa9e94e00b8a8ab3a5661bfc549f7cbe81d29350356da0 WatchSource:0}: Error finding container 0d837ec5ab4d957b09fa9e94e00b8a8ab3a5661bfc549f7cbe81d29350356da0: Status 404 returned error can't find the container with id 0d837ec5ab4d957b09fa9e94e00b8a8ab3a5661bfc549f7cbe81d29350356da0
Sep 19 08:43:33.165236 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:33.165208    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerStarted","Data":"0d837ec5ab4d957b09fa9e94e00b8a8ab3a5661bfc549f7cbe81d29350356da0"}
Sep 19 08:43:34.171187 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:34.171159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerStarted","Data":"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"}
Sep 19 08:43:55.275347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:55.275312    2584 generic.go:356] "Generic (PLEG): container finished" podID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerID="0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b" exitCode=0
Sep 19 08:43:55.275781 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:55.275383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerDied","Data":"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"}
Sep 19 08:43:56.281796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:56.281762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerStarted","Data":"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"}
Sep 19 08:43:56.282184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:56.281838    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerStarted","Data":"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"}
Sep 19 08:43:56.282184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:56.282137    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:43:56.302203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:56.302159    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" podStartSLOduration=4.223242891 podStartE2EDuration="25.302146397s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.078903506s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:56.299566081 +0000 UTC m=+1404.384066883" watchObservedRunningTime="2026-09-19 08:43:56.302146397 +0000 UTC m=+1404.386647199"
Sep 19 08:43:57.286540 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:43:57.286507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:44:03.297632 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:44:03.297606    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:45:25.933691 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:25.933659    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:45:33.995460 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:33.995429    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:45:33.998588 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:33.998565    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:45:34.001634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:34.001612    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:45:34.004533 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:34.004510    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:45:36.943229 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:36.943182    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 08:45:37.737715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:37.737680    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:45:37.737995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:37.737956    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kserve-container" containerID="cri-o://0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a" gracePeriod=30
Sep 19 08:45:37.738090 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:37.738040    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kube-rbac-proxy" containerID="cri-o://098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6" gracePeriod=30
Sep 19 08:45:38.291729 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:38.291691    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 19 08:45:38.806256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:38.806222    2584 generic.go:356] "Generic (PLEG): container finished" podID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerID="098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6" exitCode=2
Sep 19 08:45:38.806426 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:38.806266    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerDied","Data":"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"}
Sep 19 08:45:39.085340 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.085320    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:45:39.131326 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131301    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config\") pod \"e31c3831-d271-4db8-889e-bfe5f87224ba\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") "
Sep 19 08:45:39.131433 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131333    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location\") pod \"e31c3831-d271-4db8-889e-bfe5f87224ba\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") "
Sep 19 08:45:39.131433 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert\") pod \"e31c3831-d271-4db8-889e-bfe5f87224ba\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") "
Sep 19 08:45:39.131503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131472    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") pod \"e31c3831-d271-4db8-889e-bfe5f87224ba\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") "
Sep 19 08:45:39.131567 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8c52\" (UniqueName: \"kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52\") pod \"e31c3831-d271-4db8-889e-bfe5f87224ba\" (UID: \"e31c3831-d271-4db8-889e-bfe5f87224ba\") "
Sep 19 08:45:39.131615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131592    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e31c3831-d271-4db8-889e-bfe5f87224ba" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:45:39.131703 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131682    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e31c3831-d271-4db8-889e-bfe5f87224ba" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:45:39.131820 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131806    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:45:39.131881 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.131821    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e31c3831-d271-4db8-889e-bfe5f87224ba-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:45:39.133441 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.133420    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e31c3831-d271-4db8-889e-bfe5f87224ba" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:45:39.133512 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.133476    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52" (OuterVolumeSpecName: "kube-api-access-r8c52") pod "e31c3831-d271-4db8-889e-bfe5f87224ba" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba"). InnerVolumeSpecName "kube-api-access-r8c52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:45:39.135316 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.135292    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e31c3831-d271-4db8-889e-bfe5f87224ba" (UID: "e31c3831-d271-4db8-889e-bfe5f87224ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:45:39.232636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.232612    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8c52\" (UniqueName: \"kubernetes.io/projected/e31c3831-d271-4db8-889e-bfe5f87224ba-kube-api-access-r8c52\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:45:39.232636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.232635    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e31c3831-d271-4db8-889e-bfe5f87224ba-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:45:39.232785 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.232646    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e31c3831-d271-4db8-889e-bfe5f87224ba-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:45:39.812753 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.812718    2584 generic.go:356] "Generic (PLEG): container finished" podID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerID="0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a" exitCode=0
Sep 19 08:45:39.813171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.812776    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"
Sep 19 08:45:39.813171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.812796    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerDied","Data":"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"}
Sep 19 08:45:39.813171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.812853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq" event={"ID":"e31c3831-d271-4db8-889e-bfe5f87224ba","Type":"ContainerDied","Data":"0d837ec5ab4d957b09fa9e94e00b8a8ab3a5661bfc549f7cbe81d29350356da0"}
Sep 19 08:45:39.813171 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.812881    2584 scope.go:117] "RemoveContainer" containerID="098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"
Sep 19 08:45:39.821877 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.821861    2584 scope.go:117] "RemoveContainer" containerID="0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"
Sep 19 08:45:39.828937 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.828922    2584 scope.go:117] "RemoveContainer" containerID="0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"
Sep 19 08:45:39.834352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.834331    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:45:39.836413 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.836396    2584 scope.go:117] "RemoveContainer" containerID="098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"
Sep 19 08:45:39.836676 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:45:39.836657    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6\": container with ID starting with 098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6 not found: ID does not exist" containerID="098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"
Sep 19 08:45:39.836745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.836683    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6"} err="failed to get container status \"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6\": rpc error: code = NotFound desc = could not find container \"098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6\": container with ID starting with 098bc2c9b360c21531a2b3ee2422108697c34606db1c2b7f734bf03ab39cb0c6 not found: ID does not exist"
Sep 19 08:45:39.836745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.836699    2584 scope.go:117] "RemoveContainer" containerID="0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"
Sep 19 08:45:39.836952 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:45:39.836933    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a\": container with ID starting with 0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a not found: ID does not exist" containerID="0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"
Sep 19 08:45:39.837047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.836961    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a"} err="failed to get container status \"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a\": rpc error: code = NotFound desc = could not find container \"0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a\": container with ID starting with 0612f2a7dccba3986ce77d20d075816f1b8f75c89a26166ffd12f982d3d50a4a not found: ID does not exist"
Sep 19 08:45:39.837047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.836998    2584 scope.go:117] "RemoveContainer" containerID="0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"
Sep 19 08:45:39.837251 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:45:39.837235    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b\": container with ID starting with 0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b not found: ID does not exist" containerID="0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"
Sep 19 08:45:39.837302 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.837257    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b"} err="failed to get container status \"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b\": rpc error: code = NotFound desc = could not find container \"0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b\": container with ID starting with 0990c5fe29dffcf223248b1229d7a3aeaf04d402d9309bab843f6149a18fc24b not found: ID does not exist"
Sep 19 08:45:39.840681 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.840661    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sgmzq"]
Sep 19 08:45:39.906390 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:39.906368    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" path="/var/lib/kubelet/pods/e31c3831-d271-4db8-889e-bfe5f87224ba/volumes"
Sep 19 08:45:42.923867 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.923840    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:45:42.924243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924216    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kserve-container"
Sep 19 08:45:42.924243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924229    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kserve-container"
Sep 19 08:45:42.924243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924243    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="storage-initializer"
Sep 19 08:45:42.924350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924250    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="storage-initializer"
Sep 19 08:45:42.924350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924260    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kube-rbac-proxy"
Sep 19 08:45:42.924350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924266    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kube-rbac-proxy"
Sep 19 08:45:42.924350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924328    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kube-rbac-proxy"
Sep 19 08:45:42.924350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.924336    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e31c3831-d271-4db8-889e-bfe5f87224ba" containerName="kserve-container"
Sep 19 08:45:42.926493 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.926476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:42.929019 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.928999    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:45:42.929019 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929015    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:45:42.929268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929222    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:45:42.929268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929252    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 19 08:45:42.929397 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929308    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:45:42.929397 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929320    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:45:42.930009 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.929993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 19 08:45:42.937686 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.937668    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:45:42.958527 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.958502    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:42.958636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.958550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:42.958636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.958574    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:42.958636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.958608    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:42.958796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:42.958640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66qj\" (UniqueName: \"kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.059771 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.059747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66qj\" (UniqueName: \"kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.059931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.059788    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.059931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.059820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.059931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.059836    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.059931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.059860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.060177 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:45:43.059993    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 19 08:45:43.060177 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:45:43.060066    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls podName:47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3 nodeName:}" failed. No retries permitted until 2026-09-19 08:45:43.560047425 +0000 UTC m=+1511.644548211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 19 08:45:43.060307 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.060283    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.060515 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.060501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.060577 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.060544    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.070023 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.070000    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66qj\" (UniqueName: \"kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.564587 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.564561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.566899 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.566880    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:43.837799 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:43.837742    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:45:44.113000 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:44.112963    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:45:44.114757 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:45:44.114727    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fdfefa_7fab_489e_9c3c_6abe1d3ce1d3.slice/crio-e99fb126c2f3fd64c367610386e1b9ed05db1879d7687ecd43095061b060707a WatchSource:0}: Error finding container e99fb126c2f3fd64c367610386e1b9ed05db1879d7687ecd43095061b060707a: Status 404 returned error can't find the container with id e99fb126c2f3fd64c367610386e1b9ed05db1879d7687ecd43095061b060707a
Sep 19 08:45:44.842977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:44.842941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerStarted","Data":"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"}
Sep 19 08:45:44.843182 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:45:44.843046    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerStarted","Data":"e99fb126c2f3fd64c367610386e1b9ed05db1879d7687ecd43095061b060707a"}
Sep 19 08:46:03.944900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:03.944868    2584 generic.go:356] "Generic (PLEG): container finished" podID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerID="4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12" exitCode=0
Sep 19 08:46:03.945329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:03.944949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerDied","Data":"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"}
Sep 19 08:46:04.954178 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:04.954144    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerStarted","Data":"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"}
Sep 19 08:46:04.954541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:04.954206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerStarted","Data":"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"}
Sep 19 08:46:04.954541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:04.954393    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:46:04.954541 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:04.954435    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:46:04.975491 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:04.975439    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" podStartSLOduration=3.7367862609999998 podStartE2EDuration="22.975426658s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.238640397s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:46:04.97381747 +0000 UTC m=+1533.058318287" watchObservedRunningTime="2026-09-19 08:46:04.975426658 +0000 UTC m=+1533.059927456"
Sep 19 08:46:10.964517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:46:10.964484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:47:33.939424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:33.939345    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:47:44.946196 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:44.946146    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 19 08:47:49.404366 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:49.404322    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:47:49.404922 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:49.404652    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kserve-container" containerID="cri-o://6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48" gracePeriod=30
Sep 19 08:47:49.404922 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:49.404715    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kube-rbac-proxy" containerID="cri-o://ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0" gracePeriod=30
Sep 19 08:47:50.451868 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.451836    2584 generic.go:356] "Generic (PLEG): container finished" podID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerID="ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0" exitCode=2
Sep 19 08:47:50.451868 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.451873    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerDied","Data":"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"}
Sep 19 08:47:50.750719 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.750694    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:47:50.918595 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918566    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config\") pod \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") "
Sep 19 08:47:50.918764 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918652    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert\") pod \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") "
Sep 19 08:47:50.918764 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918743    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location\") pod \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") "
Sep 19 08:47:50.918844 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918781    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") pod \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") "
Sep 19 08:47:50.918844 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918836    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h66qj\" (UniqueName: \"kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj\") pod \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\" (UID: \"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3\") "
Sep 19 08:47:50.918928 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.918906    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:47:50.919097 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.919067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:47:50.919193 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.919154    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:47:50.919193 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.919164    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:47:50.920909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.920878    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:47:50.921262 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.921244    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj" (OuterVolumeSpecName: "kube-api-access-h66qj") pod "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3"). InnerVolumeSpecName "kube-api-access-h66qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:47:50.923763 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:50.923741    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" (UID: "47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:51.019845 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.019791    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:47:51.019845 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.019826    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:47:51.019968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.019846    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66qj\" (UniqueName: \"kubernetes.io/projected/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3-kube-api-access-h66qj\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:47:51.458157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.458086    2584 generic.go:356] "Generic (PLEG): container finished" podID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerID="6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48" exitCode=0
Sep 19 08:47:51.458157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.458150    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"
Sep 19 08:47:51.458544 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.458167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerDied","Data":"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"}
Sep 19 08:47:51.458544 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.458229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc" event={"ID":"47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3","Type":"ContainerDied","Data":"e99fb126c2f3fd64c367610386e1b9ed05db1879d7687ecd43095061b060707a"}
Sep 19 08:47:51.458544 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.458257    2584 scope.go:117] "RemoveContainer" containerID="ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"
Sep 19 08:47:51.468338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.468320    2584 scope.go:117] "RemoveContainer" containerID="6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"
Sep 19 08:47:51.476032 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.476009    2584 scope.go:117] "RemoveContainer" containerID="4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"
Sep 19 08:47:51.482018 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.481996    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:47:51.484291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.484275    2584 scope.go:117] "RemoveContainer" containerID="ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"
Sep 19 08:47:51.484576 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:47:51.484555    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0\": container with ID starting with ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0 not found: ID does not exist" containerID="ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"
Sep 19 08:47:51.484643 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.484588    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0"} err="failed to get container status \"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0\": rpc error: code = NotFound desc = could not find container \"ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0\": container with ID starting with ee185a91e8ae87ab2dc1f9c14cfa40a1ee5f718d7633f790c38b843c8915a9f0 not found: ID does not exist"
Sep 19 08:47:51.484643 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.484611    2584 scope.go:117] "RemoveContainer" containerID="6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"
Sep 19 08:47:51.484875 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:47:51.484847    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48\": container with ID starting with 6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48 not found: ID does not exist" containerID="6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"
Sep 19 08:47:51.484917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.484881    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48"} err="failed to get container status \"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48\": rpc error: code = NotFound desc = could not find container \"6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48\": container with ID starting with 6c87c4efb01cd0c98e5d1f9ab2d3c89acb71e1ff3a777ba49e676fae3c618c48 not found: ID does not exist"
Sep 19 08:47:51.484917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.484896    2584 scope.go:117] "RemoveContainer" containerID="4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"
Sep 19 08:47:51.485170 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:47:51.485152    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12\": container with ID starting with 4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12 not found: ID does not exist" containerID="4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"
Sep 19 08:47:51.485239 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.485178    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12"} err="failed to get container status \"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12\": rpc error: code = NotFound desc = could not find container \"4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12\": container with ID starting with 4dd0098e21690e7367056c17d24b8a6f96ab3d911f74d2353329d2d67712ac12 not found: ID does not exist"
Sep 19 08:47:51.486143 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.486127    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-4r8tc"]
Sep 19 08:47:51.908066 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:51.908040    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" path="/var/lib/kubelet/pods/47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3/volumes"
Sep 19 08:47:54.674391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674362    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:47:54.674749 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674719    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kube-rbac-proxy"
Sep 19 08:47:54.674749 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674730    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kube-rbac-proxy"
Sep 19 08:47:54.674749 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674748    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kserve-container"
Sep 19 08:47:54.674886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674754    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kserve-container"
Sep 19 08:47:54.674886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674763    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="storage-initializer"
Sep 19 08:47:54.674886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674769    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="storage-initializer"
Sep 19 08:47:54.674886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674828    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kserve-container"
Sep 19 08:47:54.674886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.674836    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fdfefa-7fab-489e-9c3c-6abe1d3ce1d3" containerName="kube-rbac-proxy"
Sep 19 08:47:54.677884 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.677868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.680564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.680544    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:47:54.680701 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.680578    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:47:54.680701 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.680656    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 19 08:47:54.680913 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.680888    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:47:54.681045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.680935    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:47:54.681045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.681009    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 19 08:47:54.681577 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.681558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:47:54.687607 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.687585    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:47:54.851005 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.850950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.851181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.851029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.851181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.851088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.851275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.851227    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxgl\" (UniqueName: \"kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.851275 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.851257    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxgl\" (UniqueName: \"kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952268 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952513 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952513 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952513 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:47:54.952422    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 19 08:47:54.952513 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:47:54.952496    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls podName:30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd nodeName:}" failed. No retries permitted until 2026-09-19 08:47:55.45247639 +0000 UTC m=+1643.536977180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 19 08:47:54.952762 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.952962 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.952943    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.953051 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.953032    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:54.961015 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:54.960998    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxgl\" (UniqueName: \"kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:55.456700 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:55.456667    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:55.459046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:55.459017    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:55.590138 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:55.590111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:47:55.860853 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:55.860831    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:47:55.862925 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:47:55.862902    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fd52a8_9bfe_41c0_8ae0_ac537d4a50dd.slice/crio-56e7ea7009508c036331fb50d3d40e5afb7dedf457db1bb6e16ee29041ff6fe2 WatchSource:0}: Error finding container 56e7ea7009508c036331fb50d3d40e5afb7dedf457db1bb6e16ee29041ff6fe2: Status 404 returned error can't find the container with id 56e7ea7009508c036331fb50d3d40e5afb7dedf457db1bb6e16ee29041ff6fe2
Sep 19 08:47:56.486242 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:56.486207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerStarted","Data":"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"}
Sep 19 08:47:56.486429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:47:56.486278    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerStarted","Data":"56e7ea7009508c036331fb50d3d40e5afb7dedf457db1bb6e16ee29041ff6fe2"}
Sep 19 08:48:16.579754 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:16.579719    2584 generic.go:356] "Generic (PLEG): container finished" podID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerID="934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39" exitCode=0
Sep 19 08:48:16.580131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:16.579786    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerDied","Data":"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"}
Sep 19 08:48:17.589288 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:17.589253    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerStarted","Data":"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"}
Sep 19 08:48:17.589717 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:17.589343    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerStarted","Data":"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"}
Sep 19 08:48:17.589717 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:17.589546    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:48:17.589717 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:17.589597    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:48:17.607669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:17.607621    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" podStartSLOduration=3.372647019 podStartE2EDuration="23.607604208s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.234957189s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:48:17.607004917 +0000 UTC m=+1665.691505720" watchObservedRunningTime="2026-09-19 08:48:17.607604208 +0000 UTC m=+1665.692105013"
Sep 19 08:48:23.601314 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:48:23.601272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:49:53.939747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:49:53.939720    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:50:00.823669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:00.823633    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:50:00.824207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:00.823863    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kserve-container" containerID="cri-o://0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de" gracePeriod=30
Sep 19 08:50:00.824207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:00.823908    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kube-rbac-proxy" containerID="cri-o://5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5" gracePeriod=30
Sep 19 08:50:01.093588 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:01.093505    2584 generic.go:356] "Generic (PLEG): container finished" podID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerID="5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5" exitCode=2
Sep 19 08:50:01.093732 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:01.093577    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerDied","Data":"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"}
Sep 19 08:50:02.279362 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.279339    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:50:02.302277 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxgl\" (UniqueName: \"kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl\") pod \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") "
Sep 19 08:50:02.302413 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302284    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert\") pod \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") "
Sep 19 08:50:02.302413 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") pod \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") "
Sep 19 08:50:02.302413 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302359    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config\") pod \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") "
Sep 19 08:50:02.302589 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302558    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location\") pod \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\" (UID: \"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd\") "
Sep 19 08:50:02.302751 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302714    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:50:02.302751 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:50:02.303042 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302882    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:02.303042 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.302903    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:02.305107 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.305066    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:50:02.305350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.305325    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl" (OuterVolumeSpecName: "kube-api-access-fxxgl") pod "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd"). InnerVolumeSpecName "kube-api-access-fxxgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:50:02.307996 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.307949    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" (UID: "30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:02.404114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.404028    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:02.404114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.404056    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:02.404114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:02.404066    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxgl\" (UniqueName: \"kubernetes.io/projected/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd-kube-api-access-fxxgl\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:03.106127 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.106092    2584 generic.go:356] "Generic (PLEG): container finished" podID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerID="0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de" exitCode=0
Sep 19 08:50:03.106323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.106155    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"
Sep 19 08:50:03.106323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.106159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerDied","Data":"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"}
Sep 19 08:50:03.106323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.106217    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9" event={"ID":"30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd","Type":"ContainerDied","Data":"56e7ea7009508c036331fb50d3d40e5afb7dedf457db1bb6e16ee29041ff6fe2"}
Sep 19 08:50:03.106323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.106244    2584 scope.go:117] "RemoveContainer" containerID="5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"
Sep 19 08:50:03.116165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.116149    2584 scope.go:117] "RemoveContainer" containerID="0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"
Sep 19 08:50:03.123778 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.123761    2584 scope.go:117] "RemoveContainer" containerID="934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"
Sep 19 08:50:03.128205 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.128183    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:50:03.131371 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.131354    2584 scope.go:117] "RemoveContainer" containerID="5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"
Sep 19 08:50:03.131731 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:03.131702    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5\": container with ID starting with 5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5 not found: ID does not exist" containerID="5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"
Sep 19 08:50:03.131897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.131864    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5"} err="failed to get container status \"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5\": rpc error: code = NotFound desc = could not find container \"5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5\": container with ID starting with 5e099052e16b09c8a7e91eb4a8d4a750b4fc78c6d53c50d28cde778240d576f5 not found: ID does not exist"
Sep 19 08:50:03.131897 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.131898    2584 scope.go:117] "RemoveContainer" containerID="0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"
Sep 19 08:50:03.132396 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:03.132368    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de\": container with ID starting with 0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de not found: ID does not exist" containerID="0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"
Sep 19 08:50:03.132469 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.132404    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de"} err="failed to get container status \"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de\": rpc error: code = NotFound desc = could not find container \"0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de\": container with ID starting with 0f4ab445befc6c3cc23d6345200ead3006240a9fd8a9eea491d568f86b3764de not found: ID does not exist"
Sep 19 08:50:03.132469 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.132427    2584 scope.go:117] "RemoveContainer" containerID="934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"
Sep 19 08:50:03.132697 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:03.132679    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39\": container with ID starting with 934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39 not found: ID does not exist" containerID="934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"
Sep 19 08:50:03.132739 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.132703    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39"} err="failed to get container status \"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39\": rpc error: code = NotFound desc = could not find container \"934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39\": container with ID starting with 934ab4f93fc0480556d4b42d8925be68d3fd2d3ab96e1f8568a26eaf3e8fcd39 not found: ID does not exist"
Sep 19 08:50:03.133864 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.133846    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-gspm9"]
Sep 19 08:50:03.906709 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:03.906679    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" path="/var/lib/kubelet/pods/30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd/volumes"
Sep 19 08:50:06.008902 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.008864    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:06.009529 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009512    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kserve-container"
Sep 19 08:50:06.009605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009532    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kserve-container"
Sep 19 08:50:06.009605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009557    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="storage-initializer"
Sep 19 08:50:06.009605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009566    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="storage-initializer"
Sep 19 08:50:06.009605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009582    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kube-rbac-proxy"
Sep 19 08:50:06.009605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009591    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kube-rbac-proxy"
Sep 19 08:50:06.009806 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009670    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kube-rbac-proxy"
Sep 19 08:50:06.009806 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.009685    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="30fd52a8-9bfe-41c0-8ae0-ac537d4a50dd" containerName="kserve-container"
Sep 19 08:50:06.015225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.015204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.017526 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 19 08:50:06.017656 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:50:06.017656 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017634    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:50:06.017656 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017640    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:50:06.017834 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017639    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:50:06.017834 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.017703    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:50:06.018399 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.018379    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 19 08:50:06.022396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.022375    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:06.034373 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.034338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.034505 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.034406    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r2x\" (UniqueName: \"kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.034505 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.034476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.034648 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.034530    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.034648 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.034580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.135741 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.135711    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r2x\" (UniqueName: \"kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.135880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.135758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.135880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.135794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.135880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.135820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.135880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.135863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.136291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.136267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.136538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.136519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.136571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.136554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.138356 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.138332    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.147292 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.147269    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r2x\" (UniqueName: \"kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.327802 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.327717    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:06.599444 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:06.599420    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:06.601875 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:50:06.601843    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2be84e_776c_4b2c_af7b_d56c5fa82f9e.slice/crio-6681229c122a538ad2981157fbf1f4ff185044c543b886e8ac6ce77948dd4410 WatchSource:0}: Error finding container 6681229c122a538ad2981157fbf1f4ff185044c543b886e8ac6ce77948dd4410: Status 404 returned error can't find the container with id 6681229c122a538ad2981157fbf1f4ff185044c543b886e8ac6ce77948dd4410
Sep 19 08:50:07.131446 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:07.131396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerStarted","Data":"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"}
Sep 19 08:50:07.131846 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:07.131487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerStarted","Data":"6681229c122a538ad2981157fbf1f4ff185044c543b886e8ac6ce77948dd4410"}
Sep 19 08:50:16.175886 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:16.175850    2584 generic.go:356] "Generic (PLEG): container finished" podID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerID="a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef" exitCode=0
Sep 19 08:50:16.176282 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:16.175925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerDied","Data":"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"}
Sep 19 08:50:17.183619 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:17.183587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerStarted","Data":"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"}
Sep 19 08:50:17.183999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:17.183648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerStarted","Data":"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"}
Sep 19 08:50:17.183999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:17.183832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:17.204461 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:17.204415    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podStartSLOduration=3.210343657 podStartE2EDuration="12.204402079s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.994058422s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:50:17.203319292 +0000 UTC m=+1785.287820105" watchObservedRunningTime="2026-09-19 08:50:17.204402079 +0000 UTC m=+1785.288902882"
Sep 19 08:50:18.188384 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:18.188346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:18.189648 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:18.189615    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 19 08:50:19.194082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:19.194046    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 19 08:50:24.198405 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:24.198371    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:24.198999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:24.198952    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 19 08:50:34.029100 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:34.029071    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:50:34.031950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:34.031928    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:50:34.034608 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:34.034582    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:50:34.037159 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:34.037144    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:50:34.199821 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:34.199800    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:48.385434 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:48.385393    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:48.385883 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:48.385682    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container" containerID="cri-o://0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792" gracePeriod=30
Sep 19 08:50:48.385883 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:48.385764    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kube-rbac-proxy" containerID="cri-o://3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a" gracePeriod=30
Sep 19 08:50:49.194301 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:49.194252    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 19 08:50:49.359308 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:49.359273    2584 generic.go:356] "Generic (PLEG): container finished" podID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerID="3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a" exitCode=2
Sep 19 08:50:49.359471 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:49.359316    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerDied","Data":"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"}
Sep 19 08:50:50.032204 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.032179    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:50.119545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119471    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert\") pod \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") "
Sep 19 08:50:50.119545 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls\") pod \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") "
Sep 19 08:50:50.119765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119545    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57r2x\" (UniqueName: \"kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x\") pod \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") "
Sep 19 08:50:50.119765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119582    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location\") pod \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") "
Sep 19 08:50:50.119765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119654    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config\") pod \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\" (UID: \"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e\") "
Sep 19 08:50:50.119936 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.119826    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" (UID: "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:50:50.120154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.120119    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" (UID: "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:50:50.120647 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.120621    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" (UID: "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:50.121549 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.121525    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" (UID: "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:50:50.121707 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.121686    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x" (OuterVolumeSpecName: "kube-api-access-57r2x") pod "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" (UID: "fb2be84e-776c-4b2c-af7b-d56c5fa82f9e"). InnerVolumeSpecName "kube-api-access-57r2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:50:50.220358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.220333    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:50.220358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.220354    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:50.220488 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.220364    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:50.220488 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.220373    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:50.220488 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.220381    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57r2x\" (UniqueName: \"kubernetes.io/projected/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e-kube-api-access-57r2x\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:50:50.367016 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.366978    2584 generic.go:356] "Generic (PLEG): container finished" podID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerID="0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792" exitCode=0
Sep 19 08:50:50.367119 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.367051    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"
Sep 19 08:50:50.367165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.367055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerDied","Data":"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"}
Sep 19 08:50:50.367207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.367175    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh" event={"ID":"fb2be84e-776c-4b2c-af7b-d56c5fa82f9e","Type":"ContainerDied","Data":"6681229c122a538ad2981157fbf1f4ff185044c543b886e8ac6ce77948dd4410"}
Sep 19 08:50:50.367207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.367201    2584 scope.go:117] "RemoveContainer" containerID="3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"
Sep 19 08:50:50.376274 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.376259    2584 scope.go:117] "RemoveContainer" containerID="0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"
Sep 19 08:50:50.383609 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.383592    2584 scope.go:117] "RemoveContainer" containerID="a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"
Sep 19 08:50:50.389057 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.389036    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:50.394654 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.393225    2584 scope.go:117] "RemoveContainer" containerID="3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"
Sep 19 08:50:50.394654 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.393406    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-kpnqh"]
Sep 19 08:50:50.395233 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:50.395211    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a\": container with ID starting with 3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a not found: ID does not exist" containerID="3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"
Sep 19 08:50:50.395314 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.395237    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a"} err="failed to get container status \"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a\": rpc error: code = NotFound desc = could not find container \"3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a\": container with ID starting with 3b81dd7265f5f4cbe6bfdf714147518e271e21083e4039f569349b4f2189df7a not found: ID does not exist"
Sep 19 08:50:50.395314 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.395254    2584 scope.go:117] "RemoveContainer" containerID="0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"
Sep 19 08:50:50.395570 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:50.395545    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792\": container with ID starting with 0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792 not found: ID does not exist" containerID="0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"
Sep 19 08:50:50.395649 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.395580    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792"} err="failed to get container status \"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792\": rpc error: code = NotFound desc = could not find container \"0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792\": container with ID starting with 0e283a46a56e91dd9530f4e8d6dce7b0cf3a443a098c5352651fd2cc104e1792 not found: ID does not exist"
Sep 19 08:50:50.395649 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.395603    2584 scope.go:117] "RemoveContainer" containerID="a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"
Sep 19 08:50:50.395858 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:50.395842    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef\": container with ID starting with a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef not found: ID does not exist" containerID="a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"
Sep 19 08:50:50.395901 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:50.395862    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef"} err="failed to get container status \"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef\": rpc error: code = NotFound desc = could not find container \"a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef\": container with ID starting with a02f8e636b2a45b7825c4f1dd16e023a305f215d034cd4c514bf53d64bbd89ef not found: ID does not exist"
Sep 19 08:50:51.908001 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:51.907945    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" path="/var/lib/kubelet/pods/fb2be84e-776c-4b2c-af7b-d56c5fa82f9e/volumes"
Sep 19 08:50:53.587445 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587408    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:50:53.587808 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587794    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kube-rbac-proxy"
Sep 19 08:50:53.587862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587809    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kube-rbac-proxy"
Sep 19 08:50:53.587862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587823    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="storage-initializer"
Sep 19 08:50:53.587862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587829    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="storage-initializer"
Sep 19 08:50:53.587862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587851    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container"
Sep 19 08:50:53.587862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587858    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container"
Sep 19 08:50:53.588045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587918    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kube-rbac-proxy"
Sep 19 08:50:53.588045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.587930    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb2be84e-776c-4b2c-af7b-d56c5fa82f9e" containerName="kserve-container"
Sep 19 08:50:53.591299 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.591283    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.593796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593770    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 19 08:50:53.593944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593790    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:50:53.593944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 19 08:50:53.593944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:50:53.594182 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:50:53.594182 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.593968    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:50:53.594381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.594364    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:50:53.602674 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.602651    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:50:53.646318 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.646291    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.646419 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.646326    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.646419 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.646355    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.646419 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.646400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdhz\" (UniqueName: \"kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.646529 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.646436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747188 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747094    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747188 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747188 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdhz\" (UniqueName: \"kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747309 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747207    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747309 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:53.747221    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 19 08:50:53.747309 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:50:53.747292    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls podName:d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295 nodeName:}" failed. No retries permitted until 2026-09-19 08:50:54.24727258 +0000 UTC m=+1822.331773373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 19 08:50:53.747492 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747858 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747839    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.747901 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.747858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:53.756766 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:53.756738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdhz\" (UniqueName: \"kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:54.251107 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:54.251073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:54.253412 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:54.253393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:54.501914 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:54.501829    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:50:54.772427 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:54.772397    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:50:54.774331 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:50:54.774303    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a63fc0_b817_4f96_9a7f_7cdf3b9dd295.slice/crio-5b5c76f71b12fef11173776205eba1ff395648187a6aa47611100c36c8925f46 WatchSource:0}: Error finding container 5b5c76f71b12fef11173776205eba1ff395648187a6aa47611100c36c8925f46: Status 404 returned error can't find the container with id 5b5c76f71b12fef11173776205eba1ff395648187a6aa47611100c36c8925f46
Sep 19 08:50:55.393764 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:55.393724    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerStarted","Data":"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"}
Sep 19 08:50:55.393942 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:50:55.393802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerStarted","Data":"5b5c76f71b12fef11173776205eba1ff395648187a6aa47611100c36c8925f46"}
Sep 19 08:51:00.421265 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:00.421231    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerID="751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0" exitCode=0
Sep 19 08:51:00.421682 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:00.421304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerDied","Data":"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"}
Sep 19 08:51:01.432614 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:01.432578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerStarted","Data":"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"}
Sep 19 08:51:01.433129 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:01.432653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerStarted","Data":"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"}
Sep 19 08:51:01.433129 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:01.432876    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:01.452164 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:01.452112    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podStartSLOduration=3.856933422 podStartE2EDuration="8.45209852s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.595165098s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:01.450419847 +0000 UTC m=+1829.534920654" watchObservedRunningTime="2026-09-19 08:51:01.45209852 +0000 UTC m=+1829.536599323"
Sep 19 08:51:02.437899 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:02.437867    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:02.439532 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:02.439500    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 19 08:51:03.451021 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:03.450957    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 19 08:51:08.455170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:08.455141    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:08.455670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:08.455579    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 19 08:51:18.457103 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:18.457071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:28.924849 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:28.924814    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:51:28.925289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:28.925088    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container" containerID="cri-o://ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4" gracePeriod=30
Sep 19 08:51:28.925289 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:28.925131    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kube-rbac-proxy" containerID="cri-o://0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2" gracePeriod=30
Sep 19 08:51:29.583624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:29.583591    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerID="0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2" exitCode=2
Sep 19 08:51:29.583804 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:29.583671    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerDied","Data":"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"}
Sep 19 08:51:30.268941 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.268919    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:30.356235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356143    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") pod \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") "
Sep 19 08:51:30.356235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356186    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pdhz\" (UniqueName: \"kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz\") pod \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") "
Sep 19 08:51:30.356235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356215    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location\") pod \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") "
Sep 19 08:51:30.356524 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356270    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config\") pod \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") "
Sep 19 08:51:30.356524 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356347    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert\") pod \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\" (UID: \"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295\") "
Sep 19 08:51:30.356799 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:51:30.356799 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.356760    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:51:30.357533 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.357505    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:30.358302 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.358280    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz" (OuterVolumeSpecName: "kube-api-access-7pdhz") pod "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295"). InnerVolumeSpecName "kube-api-access-7pdhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:51:30.358446 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.358426    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" (UID: "d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:51:30.457611 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.457580    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdhz\" (UniqueName: \"kubernetes.io/projected/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-api-access-7pdhz\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:51:30.457611 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.457607    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:51:30.457611 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.457616    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:51:30.457830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.457625    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:51:30.457830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.457634    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:51:30.590606 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.590571    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerID="ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4" exitCode=0
Sep 19 08:51:30.590768 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.590628    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"
Sep 19 08:51:30.590768 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.590662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerDied","Data":"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"}
Sep 19 08:51:30.590768 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.590736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd" event={"ID":"d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295","Type":"ContainerDied","Data":"5b5c76f71b12fef11173776205eba1ff395648187a6aa47611100c36c8925f46"}
Sep 19 08:51:30.590872 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.590775    2584 scope.go:117] "RemoveContainer" containerID="0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"
Sep 19 08:51:30.599830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.599812    2584 scope.go:117] "RemoveContainer" containerID="ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"
Sep 19 08:51:30.607424 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.607401    2584 scope.go:117] "RemoveContainer" containerID="751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"
Sep 19 08:51:30.613161 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.613140    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:51:30.615248 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.615215    2584 scope.go:117] "RemoveContainer" containerID="0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"
Sep 19 08:51:30.615478 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:51:30.615462    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2\": container with ID starting with 0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2 not found: ID does not exist" containerID="0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"
Sep 19 08:51:30.615528 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.615486    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2"} err="failed to get container status \"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2\": rpc error: code = NotFound desc = could not find container \"0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2\": container with ID starting with 0fd4fb49fdc287da7840854e49dd3e5912c627024d649a919de427ae58b36fb2 not found: ID does not exist"
Sep 19 08:51:30.615528 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.615502    2584 scope.go:117] "RemoveContainer" containerID="ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"
Sep 19 08:51:30.615703 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:51:30.615679    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4\": container with ID starting with ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4 not found: ID does not exist" containerID="ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"
Sep 19 08:51:30.615747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.615699    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4"} err="failed to get container status \"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4\": rpc error: code = NotFound desc = could not find container \"ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4\": container with ID starting with ba2814e3b8cab57aef4101d44df611f9c3d541d4fb58730c88ecc5343d2aabe4 not found: ID does not exist"
Sep 19 08:51:30.615747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.615711    2584 scope.go:117] "RemoveContainer" containerID="751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"
Sep 19 08:51:30.615999 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:51:30.615952    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0\": container with ID starting with 751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0 not found: ID does not exist" containerID="751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"
Sep 19 08:51:30.616046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.616033    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0"} err="failed to get container status \"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0\": rpc error: code = NotFound desc = could not find container \"751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0\": container with ID starting with 751fed53ec7a771d155a7e28f82aa7f060bc45aa2b8dab4659ea6d3cda1c6da0 not found: ID does not exist"
Sep 19 08:51:30.618870 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:30.618849    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4qzjd"]
Sep 19 08:51:31.907061 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:31.907032    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" path="/var/lib/kubelet/pods/d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295/volumes"
Sep 19 08:51:34.201879 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.201841    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:51:34.202449 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202433    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="storage-initializer"
Sep 19 08:51:34.202503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202452    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="storage-initializer"
Sep 19 08:51:34.202503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202473    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kube-rbac-proxy"
Sep 19 08:51:34.202503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202481    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kube-rbac-proxy"
Sep 19 08:51:34.202503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202493    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container"
Sep 19 08:51:34.202503 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202503    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container"
Sep 19 08:51:34.202685 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202595    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kserve-container"
Sep 19 08:51:34.202685 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.202610    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a63fc0-b817-4f96-9a7f-7cdf3b9dd295" containerName="kube-rbac-proxy"
Sep 19 08:51:34.206688 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.206668    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.209350 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.209327    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:51:34.209496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.209348    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 19 08:51:34.209496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.209357    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 19 08:51:34.209496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.209330    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:51:34.209704 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.209688    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:51:34.210189 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.210172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:51:34.210345 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.210330    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:51:34.216504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.216416    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:51:34.293388 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.293350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.293569 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.293423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.293569 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.293527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.293652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.293572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.293652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.293601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mrfv\" (UniqueName: \"kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.394292 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.394180    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.394292 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.394245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.394292 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.394282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.394620 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.394308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mrfv\" (UniqueName: \"kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.394620 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.394366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.395329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.395302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.395472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.395395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.395472 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.395403    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.398821 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.398796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.402064 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.402045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mrfv\" (UniqueName: \"kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv\") pod \"isvc-canary-create-predictor-76b74f4d6c-kvp4c\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.519196 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.519165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:34.794418 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:34.794392    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:51:34.796200 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:51:34.796173    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8293ba_28cd_450e_94b5_00469373a92a.slice/crio-2f56fc0e0150657b8663006d9ede06a0c84f1f97cd4cf531d1e15956d823755d WatchSource:0}: Error finding container 2f56fc0e0150657b8663006d9ede06a0c84f1f97cd4cf531d1e15956d823755d: Status 404 returned error can't find the container with id 2f56fc0e0150657b8663006d9ede06a0c84f1f97cd4cf531d1e15956d823755d
Sep 19 08:51:35.617198 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:35.617153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerStarted","Data":"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"}
Sep 19 08:51:35.617610 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:35.617220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerStarted","Data":"2f56fc0e0150657b8663006d9ede06a0c84f1f97cd4cf531d1e15956d823755d"}
Sep 19 08:51:38.634039 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:38.634008    2584 generic.go:356] "Generic (PLEG): container finished" podID="8f8293ba-28cd-450e-94b5-00469373a92a" containerID="be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8" exitCode=0
Sep 19 08:51:38.634351 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:38.634076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerDied","Data":"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"}
Sep 19 08:51:38.635235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:38.635217    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:51:52.726010 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:52.725958    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerStarted","Data":"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"}
Sep 19 08:51:52.726392 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:52.726038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerStarted","Data":"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"}
Sep 19 08:51:52.726392 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:52.726228    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:52.745249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:52.745189    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podStartSLOduration=1.870865607 podStartE2EDuration="18.745172101s" totalImagesPullingTime="13.141752294s" totalInitContainerRuntime="3.7325542s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:52.744371513 +0000 UTC m=+1880.828872317" watchObservedRunningTime="2026-09-19 08:51:52.745172101 +0000 UTC m=+1880.829672906"
Sep 19 08:51:53.731052 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:53.731020    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:53.732318 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:53.732289    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:51:54.735608 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:54.735566    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:51:59.739823 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:59.739788    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:51:59.740341 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:51:59.740316    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:09.740418 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:09.740382    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:19.740852 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:19.740813    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:29.741205 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:29.741167    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:39.740731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:39.740689    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:49.740530 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:49.740487    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:52:59.740440 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:52:59.740398    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 19 08:53:09.741148 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:09.741111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:53:14.184759 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:14.184721    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:53:14.185256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:14.185054    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container" containerID="cri-o://60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7" gracePeriod=30
Sep 19 08:53:14.185256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:14.185062    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kube-rbac-proxy" containerID="cri-o://13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0" gracePeriod=30
Sep 19 08:53:14.736468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:14.736429    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 19 08:53:15.129347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:15.129259    2584 generic.go:356] "Generic (PLEG): container finished" podID="8f8293ba-28cd-450e-94b5-00469373a92a" containerID="13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0" exitCode=2
Sep 19 08:53:15.129347 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:15.129312    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerDied","Data":"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"}
Sep 19 08:53:18.435496 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.435475    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:53:18.550971 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.550905    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert\") pod \"8f8293ba-28cd-450e-94b5-00469373a92a\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") "
Sep 19 08:53:18.550971 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.550947    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location\") pod \"8f8293ba-28cd-450e-94b5-00469373a92a\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") "
Sep 19 08:53:18.551203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config\") pod \"8f8293ba-28cd-450e-94b5-00469373a92a\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") "
Sep 19 08:53:18.551203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551079    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls\") pod \"8f8293ba-28cd-450e-94b5-00469373a92a\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") "
Sep 19 08:53:18.551203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mrfv\" (UniqueName: \"kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv\") pod \"8f8293ba-28cd-450e-94b5-00469373a92a\" (UID: \"8f8293ba-28cd-450e-94b5-00469373a92a\") "
Sep 19 08:53:18.551371 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f8293ba-28cd-450e-94b5-00469373a92a" (UID: "8f8293ba-28cd-450e-94b5-00469373a92a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:18.551414 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551360    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f8293ba-28cd-450e-94b5-00469373a92a" (UID: "8f8293ba-28cd-450e-94b5-00469373a92a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:18.551414 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.551397    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f8293ba-28cd-450e-94b5-00469373a92a" (UID: "8f8293ba-28cd-450e-94b5-00469373a92a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:18.553073 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.553051    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f8293ba-28cd-450e-94b5-00469373a92a" (UID: "8f8293ba-28cd-450e-94b5-00469373a92a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:18.553157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.553136    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv" (OuterVolumeSpecName: "kube-api-access-4mrfv") pod "8f8293ba-28cd-450e-94b5-00469373a92a" (UID: "8f8293ba-28cd-450e-94b5-00469373a92a"). InnerVolumeSpecName "kube-api-access-4mrfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:18.651808 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.651781    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.651808 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.651806    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f8293ba-28cd-450e-94b5-00469373a92a-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.651950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.651822    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mrfv\" (UniqueName: \"kubernetes.io/projected/8f8293ba-28cd-450e-94b5-00469373a92a-kube-api-access-4mrfv\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.651950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.651834    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f8293ba-28cd-450e-94b5-00469373a92a-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:53:18.651950 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:18.651846    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f8293ba-28cd-450e-94b5-00469373a92a-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:53:19.150770 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.150738    2584 generic.go:356] "Generic (PLEG): container finished" podID="8f8293ba-28cd-450e-94b5-00469373a92a" containerID="60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7" exitCode=0
Sep 19 08:53:19.150931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.150774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerDied","Data":"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"}
Sep 19 08:53:19.150931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.150796    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"
Sep 19 08:53:19.150931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.150823    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c" event={"ID":"8f8293ba-28cd-450e-94b5-00469373a92a","Type":"ContainerDied","Data":"2f56fc0e0150657b8663006d9ede06a0c84f1f97cd4cf531d1e15956d823755d"}
Sep 19 08:53:19.150931 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.150848    2584 scope.go:117] "RemoveContainer" containerID="13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"
Sep 19 08:53:19.160250 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.160232    2584 scope.go:117] "RemoveContainer" containerID="60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"
Sep 19 08:53:19.168157 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.168140    2584 scope.go:117] "RemoveContainer" containerID="be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"
Sep 19 08:53:19.173318 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.173299    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:53:19.176421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.176403    2584 scope.go:117] "RemoveContainer" containerID="13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"
Sep 19 08:53:19.176636 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.176617    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-kvp4c"]
Sep 19 08:53:19.176688 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.176661    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0\": container with ID starting with 13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0 not found: ID does not exist" containerID="13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"
Sep 19 08:53:19.176731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.176697    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0"} err="failed to get container status \"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0\": rpc error: code = NotFound desc = could not find container \"13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0\": container with ID starting with 13ebf44b35dee0198dc40e52cdc609c1895247b144dccaab5f215a287ffb16b0 not found: ID does not exist"
Sep 19 08:53:19.176731 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.176714    2584 scope.go:117] "RemoveContainer" containerID="60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"
Sep 19 08:53:19.176965 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.176950    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7\": container with ID starting with 60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7 not found: ID does not exist" containerID="60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"
Sep 19 08:53:19.177034 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.176970    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7"} err="failed to get container status \"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7\": rpc error: code = NotFound desc = could not find container \"60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7\": container with ID starting with 60a7c71163876f7a3e9763524b63c3bc98c53a52e934f7ca43b9b8918439c8d7 not found: ID does not exist"
Sep 19 08:53:19.177034 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.177011    2584 scope.go:117] "RemoveContainer" containerID="be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"
Sep 19 08:53:19.177265 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.177246    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8\": container with ID starting with be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8 not found: ID does not exist" containerID="be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"
Sep 19 08:53:19.177360 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.177268    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8"} err="failed to get container status \"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8\": rpc error: code = NotFound desc = could not find container \"be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8\": container with ID starting with be0aeed8f06c2cc2fffc458d85be6d51893992ef44e55049959f84034b873ea8 not found: ID does not exist"
Sep 19 08:53:19.359145 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359112    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:53:19.359504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359491    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container"
Sep 19 08:53:19.359568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359505    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container"
Sep 19 08:53:19.359568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359518    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kube-rbac-proxy"
Sep 19 08:53:19.359568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359524    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kube-rbac-proxy"
Sep 19 08:53:19.359568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359532    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="storage-initializer"
Sep 19 08:53:19.359568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359538    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="storage-initializer"
Sep 19 08:53:19.359831 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359607    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kube-rbac-proxy"
Sep 19 08:53:19.359831 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.359617    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" containerName="kserve-container"
Sep 19 08:53:19.370536 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.370500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.373344 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373318    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 19 08:53:19.373550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373535    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:53:19.373967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373702    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:53:19.373967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373741    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:53:19.373967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373807    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:53:19.373967 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.373872    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:53:19.374267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.374243    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 19 08:53:19.375660 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.375475    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:53:19.398230 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.398201    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:53:19.402154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.402113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.404101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.404084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 19 08:53:19.410601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.410578    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:53:19.459399 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.459373    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.459690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.459404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.459690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.459486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.459690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.459533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.459690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.459614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvrz\" (UniqueName: \"kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.560475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.560573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560501    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.560573 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.560652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.560652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560604    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.560652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560634    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.560805 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.560661    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 19 08:53:19.560805 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560707    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.560805 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.560726    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls podName:1106ad51-89e3-430f-b6f0-b65ed6175d5f nodeName:}" failed. No retries permitted until 2026-09-19 08:53:20.060706918 +0000 UTC m=+1968.145207701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls") pod "isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 19 08:53:19.560964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf44\" (UniqueName: \"kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.560964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.560964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvrz\" (UniqueName: \"kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.561161 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.560996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.561360 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.561337    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.561425 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.561356    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.576131 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.576103    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvrz\" (UniqueName: \"kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:19.662207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662204    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662342 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf44\" (UniqueName: \"kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662342 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.662308    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:53:19.662418 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662418 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:53:19.662370    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls podName:500f3b73-e44f-4854-bc5e-ca13bcdc70f1 nodeName:}" failed. No retries permitted until 2026-09-19 08:53:20.162352322 +0000 UTC m=+1968.246853118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls") pod "isvc-canary-promote-predictor-6546f8697f-65djb" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:53:19.662536 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.662871 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.662852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.663022 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.663004    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.672740 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.672718    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf44\" (UniqueName: \"kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:19.906710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:19.906679    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8293ba-28cd-450e-94b5-00469373a92a" path="/var/lib/kubelet/pods/8f8293ba-28cd-450e-94b5-00469373a92a/volumes"
Sep 19 08:53:20.066267 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.066221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:20.068492 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.068468    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:20.167244 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.167219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:20.169485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.169464    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-65djb\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:20.283150 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.283123    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:20.312815 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.312789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:20.555038 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.555014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:53:20.556959 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:53:20.556934    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1106ad51_89e3_430f_b6f0_b65ed6175d5f.slice/crio-e186f5599b4c4bd63a237517f7ab9abc1f3ce0f9a7071c987b5003609a280913 WatchSource:0}: Error finding container e186f5599b4c4bd63a237517f7ab9abc1f3ce0f9a7071c987b5003609a280913: Status 404 returned error can't find the container with id e186f5599b4c4bd63a237517f7ab9abc1f3ce0f9a7071c987b5003609a280913
Sep 19 08:53:20.589233 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:20.589212    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:53:20.604148 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:53:20.604119    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500f3b73_e44f_4854_bc5e_ca13bcdc70f1.slice/crio-05de3f6326e80f5b71fe30263a8cd9d15d6f1d05917efee2b85d1cae80e93a2e WatchSource:0}: Error finding container 05de3f6326e80f5b71fe30263a8cd9d15d6f1d05917efee2b85d1cae80e93a2e: Status 404 returned error can't find the container with id 05de3f6326e80f5b71fe30263a8cd9d15d6f1d05917efee2b85d1cae80e93a2e
Sep 19 08:53:21.165052 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:21.165014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerStarted","Data":"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"}
Sep 19 08:53:21.165227 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:21.165086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerStarted","Data":"e186f5599b4c4bd63a237517f7ab9abc1f3ce0f9a7071c987b5003609a280913"}
Sep 19 08:53:21.166863 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:21.166834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerStarted","Data":"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"}
Sep 19 08:53:21.166970 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:21.166892    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerStarted","Data":"05de3f6326e80f5b71fe30263a8cd9d15d6f1d05917efee2b85d1cae80e93a2e"}
Sep 19 08:53:25.007184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.007144    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:53:25.011108 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.011085    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.013702 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.013680    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 19 08:53:25.013823 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.013719    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 19 08:53:25.023011 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.022969    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:53:25.110122 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.110096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.110225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.110133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.110225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.110174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9gk\" (UniqueName: \"kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.110225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.110200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.110335 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.110217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.189862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.189831    2584 generic.go:356] "Generic (PLEG): container finished" podID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerID="1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0" exitCode=0
Sep 19 08:53:25.190007 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.189904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerDied","Data":"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"}
Sep 19 08:53:25.191661 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.191641    2584 generic.go:356] "Generic (PLEG): container finished" podID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerID="6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e" exitCode=0
Sep 19 08:53:25.191748 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.191676    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerDied","Data":"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"}
Sep 19 08:53:25.210641 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.210615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.210789 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.210669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9gk\" (UniqueName: \"kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.210789 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.210696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.210789 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.210722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.210975 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.210805    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.211201 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.211177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.211447 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.211423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.211830 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.211805    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.213341 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.213324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.218536 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.218517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9gk\" (UniqueName: \"kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk\") pod \"isvc-sklearn-v2-predictor-655bcc46cd-7cr2v\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.322358 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.322337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:25.599896 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:25.599872    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:53:25.601970 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:53:25.601943    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe2812f_3572_4040_8776_9b8b6bb41a7c.slice/crio-19937c77afe1dc43489518e467b9b52d2ceacc58c03909646774aea37eadbaf7 WatchSource:0}: Error finding container 19937c77afe1dc43489518e467b9b52d2ceacc58c03909646774aea37eadbaf7: Status 404 returned error can't find the container with id 19937c77afe1dc43489518e467b9b52d2ceacc58c03909646774aea37eadbaf7
Sep 19 08:53:26.199322 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.199279    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerStarted","Data":"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"}
Sep 19 08:53:26.199747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.199359    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerStarted","Data":"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"}
Sep 19 08:53:26.199747 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.199572    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:26.201003 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.200965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerStarted","Data":"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"}
Sep 19 08:53:26.201128 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.201032    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerStarted","Data":"19937c77afe1dc43489518e467b9b52d2ceacc58c03909646774aea37eadbaf7"}
Sep 19 08:53:26.203291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.203270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerStarted","Data":"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"}
Sep 19 08:53:26.203391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.203329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerStarted","Data":"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"}
Sep 19 08:53:26.203623 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.203602    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:26.203693 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.203655    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:26.205054 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.205025    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:53:26.221514 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.221471    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podStartSLOduration=3.438091988 podStartE2EDuration="7.221459131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783367143s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:26.220090809 +0000 UTC m=+1974.304591613" watchObservedRunningTime="2026-09-19 08:53:26.221459131 +0000 UTC m=+1974.305959934"
Sep 19 08:53:26.256939 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:26.256895    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podStartSLOduration=3.6069033 podStartE2EDuration="7.256881035s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.649977735s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:26.25558166 +0000 UTC m=+1974.340082514" watchObservedRunningTime="2026-09-19 08:53:26.256881035 +0000 UTC m=+1974.341381839"
Sep 19 08:53:27.208670 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:27.208620    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:53:27.209121 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:27.208872    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:27.210082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:27.210041    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:53:28.212953 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:28.212913    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:53:30.223796 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:30.223757    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerID="0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba" exitCode=0
Sep 19 08:53:30.224217 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:30.223830    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerDied","Data":"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"}
Sep 19 08:53:31.231349 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.231317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerStarted","Data":"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"}
Sep 19 08:53:31.231754 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.231378    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerStarted","Data":"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"}
Sep 19 08:53:31.231754 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.231638    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:31.231843 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.231778    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:31.233175 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.233151    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:53:31.253154 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:31.253113    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podStartSLOduration=3.684234724 podStartE2EDuration="7.253099121s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.568864397s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:31.251192354 +0000 UTC m=+1979.335693160" watchObservedRunningTime="2026-09-19 08:53:31.253099121 +0000 UTC m=+1979.337599993"
Sep 19 08:53:32.213132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:32.213103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:53:32.213601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:32.213578    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:53:32.235906 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:32.235876    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:53:33.217733 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:33.217706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:53:33.218183 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:33.218158    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:53:37.241800 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:37.241771    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:53:37.242439 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:37.242401    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:53:42.214426 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:42.214383    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:53:43.218665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:43.218626    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:53:47.242657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:47.242619    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:53:52.213854 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:52.213814    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:53:53.218181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:53.218146    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:53:57.242325 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:53:57.242283    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:54:02.214443 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:02.214397    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:54:03.218824 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:03.218784    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:54:07.242929 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:07.242886    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:54:12.213657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:12.213611    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:54:13.219102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:13.219061    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:54:17.242941 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:17.242901    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:54:22.213765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:22.213726    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:54:23.218858 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:23.218822    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:54:27.242761 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:27.242720    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:54:32.213640 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:32.213600    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:54:33.218880 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:33.218844    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 19 08:54:37.242856 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:37.242814    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:54:42.214167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.214140    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:54:42.334486 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.334449    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:54:42.334716 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.334690    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container" containerID="cri-o://ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d" gracePeriod=30
Sep 19 08:54:42.334815 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.334764    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kube-rbac-proxy" containerID="cri-o://4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582" gracePeriod=30
Sep 19 08:54:42.335554 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.335145    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:54:42.598499 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.598419    2584 generic.go:356] "Generic (PLEG): container finished" podID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerID="4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582" exitCode=2
Sep 19 08:54:42.598499 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:42.598465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerDied","Data":"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"}
Sep 19 08:54:43.214102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:43.214062    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 19 08:54:46.577084 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.577061    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:54:46.620542 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.620510    2584 generic.go:356] "Generic (PLEG): container finished" podID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerID="ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d" exitCode=0
Sep 19 08:54:46.620690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.620559    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"
Sep 19 08:54:46.620690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.620595    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerDied","Data":"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"}
Sep 19 08:54:46.620690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.620653    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb" event={"ID":"500f3b73-e44f-4854-bc5e-ca13bcdc70f1","Type":"ContainerDied","Data":"05de3f6326e80f5b71fe30263a8cd9d15d6f1d05917efee2b85d1cae80e93a2e"}
Sep 19 08:54:46.620690 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.620680    2584 scope.go:117] "RemoveContainer" containerID="4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"
Sep 19 08:54:46.629329 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.629297    2584 scope.go:117] "RemoveContainer" containerID="ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"
Sep 19 08:54:46.636723 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.636709    2584 scope.go:117] "RemoveContainer" containerID="6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"
Sep 19 08:54:46.644177 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.644163    2584 scope.go:117] "RemoveContainer" containerID="4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"
Sep 19 08:54:46.644462 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:54:46.644444    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582\": container with ID starting with 4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582 not found: ID does not exist" containerID="4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"
Sep 19 08:54:46.644509 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.644470    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582"} err="failed to get container status \"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582\": rpc error: code = NotFound desc = could not find container \"4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582\": container with ID starting with 4ce98f522c7ccd24634f1f125cd8f0047640c785080369745d24828d1a23f582 not found: ID does not exist"
Sep 19 08:54:46.644509 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.644486    2584 scope.go:117] "RemoveContainer" containerID="ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"
Sep 19 08:54:46.644736 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:54:46.644717    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d\": container with ID starting with ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d not found: ID does not exist" containerID="ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"
Sep 19 08:54:46.644774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.644741    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d"} err="failed to get container status \"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d\": rpc error: code = NotFound desc = could not find container \"ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d\": container with ID starting with ec83902eeb3195fd6543a4a68cc0ef2e32c6efcb32d7b4fc6e09281e1f8a582d not found: ID does not exist"
Sep 19 08:54:46.644774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.644755    2584 scope.go:117] "RemoveContainer" containerID="6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"
Sep 19 08:54:46.645019 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:54:46.644999    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e\": container with ID starting with 6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e not found: ID does not exist" containerID="6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"
Sep 19 08:54:46.645073 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.645023    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e"} err="failed to get container status \"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e\": rpc error: code = NotFound desc = could not find container \"6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e\": container with ID starting with 6f849dd5abe6b46154b95283c094f2cc3ad3dde43f269939a01b837bd32c774e not found: ID does not exist"
Sep 19 08:54:46.725964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.725940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") pod \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") "
Sep 19 08:54:46.726093 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726040    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config\") pod \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") "
Sep 19 08:54:46.726159 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726094    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location\") pod \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") "
Sep 19 08:54:46.726159 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726132    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert\") pod \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") "
Sep 19 08:54:46.726270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726158    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjf44\" (UniqueName: \"kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44\") pod \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\" (UID: \"500f3b73-e44f-4854-bc5e-ca13bcdc70f1\") "
Sep 19 08:54:46.726452 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726429    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "500f3b73-e44f-4854-bc5e-ca13bcdc70f1" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:46.726509 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726449    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "500f3b73-e44f-4854-bc5e-ca13bcdc70f1" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:54:46.726509 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.726468    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "500f3b73-e44f-4854-bc5e-ca13bcdc70f1" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:54:46.727860 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.727828    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "500f3b73-e44f-4854-bc5e-ca13bcdc70f1" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:54:46.728204 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.728184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44" (OuterVolumeSpecName: "kube-api-access-mjf44") pod "500f3b73-e44f-4854-bc5e-ca13bcdc70f1" (UID: "500f3b73-e44f-4854-bc5e-ca13bcdc70f1"). InnerVolumeSpecName "kube-api-access-mjf44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:54:46.827482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.827459    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:54:46.827482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.827481    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:54:46.827597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.827492    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:54:46.827597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.827504    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf44\" (UniqueName: \"kubernetes.io/projected/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-kube-api-access-mjf44\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:54:46.827597 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.827513    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/500f3b73-e44f-4854-bc5e-ca13bcdc70f1-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:54:46.942613 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.942589    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:54:46.948408 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:46.948388    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-65djb"]
Sep 19 08:54:47.243618 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:47.243599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:54:47.910200 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:47.910167    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" path="/var/lib/kubelet/pods/500f3b73-e44f-4854-bc5e-ca13bcdc70f1/volumes"
Sep 19 08:54:55.730205 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:55.730163    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:54:55.730710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:55.730450    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" containerID="cri-o://80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af" gracePeriod=30
Sep 19 08:54:55.730710 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:55.730545    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kube-rbac-proxy" containerID="cri-o://12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc" gracePeriod=30
Sep 19 08:54:56.675031 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:56.674971    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerID="12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc" exitCode=2
Sep 19 08:54:56.675203 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:56.675042    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerDied","Data":"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"}
Sep 19 08:54:57.236473 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:57.236433    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 19 08:54:57.243040 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:54:57.243012    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 19 08:55:01.001339 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.001307    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:55:01.162047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.161917    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn9gk\" (UniqueName: \"kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk\") pod \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") "
Sep 19 08:55:01.162047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.161966    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config\") pod \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") "
Sep 19 08:55:01.162047 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162016    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert\") pod \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") "
Sep 19 08:55:01.162316 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls\") pod \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") "
Sep 19 08:55:01.162316 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162169    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location\") pod \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\" (UID: \"0fe2812f-3572-4040-8776-9b8b6bb41a7c\") "
Sep 19 08:55:01.162422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162364    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fe2812f-3572-4040-8776-9b8b6bb41a7c" (UID: "0fe2812f-3572-4040-8776-9b8b6bb41a7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:55:01.162482 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162425    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0fe2812f-3572-4040-8776-9b8b6bb41a7c" (UID: "0fe2812f-3572-4040-8776-9b8b6bb41a7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:55:01.162582 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162548    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fe2812f-3572-4040-8776-9b8b6bb41a7c" (UID: "0fe2812f-3572-4040-8776-9b8b6bb41a7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:55:01.162713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162644    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:55:01.162713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162665    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:55:01.162713 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.162680    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fe2812f-3572-4040-8776-9b8b6bb41a7c-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:55:01.164382 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.164355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fe2812f-3572-4040-8776-9b8b6bb41a7c" (UID: "0fe2812f-3572-4040-8776-9b8b6bb41a7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:55:01.164621 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.164601    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk" (OuterVolumeSpecName: "kube-api-access-hn9gk") pod "0fe2812f-3572-4040-8776-9b8b6bb41a7c" (UID: "0fe2812f-3572-4040-8776-9b8b6bb41a7c"). InnerVolumeSpecName "kube-api-access-hn9gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:55:01.264179 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.264135    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fe2812f-3572-4040-8776-9b8b6bb41a7c-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:55:01.264179 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.264173    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9gk\" (UniqueName: \"kubernetes.io/projected/0fe2812f-3572-4040-8776-9b8b6bb41a7c-kube-api-access-hn9gk\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:55:01.703291 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.703255    2584 generic.go:356] "Generic (PLEG): container finished" podID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerID="80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af" exitCode=0
Sep 19 08:55:01.703474 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.703317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerDied","Data":"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"}
Sep 19 08:55:01.703474 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.703363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v" event={"ID":"0fe2812f-3572-4040-8776-9b8b6bb41a7c","Type":"ContainerDied","Data":"19937c77afe1dc43489518e467b9b52d2ceacc58c03909646774aea37eadbaf7"}
Sep 19 08:55:01.703474 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.703319    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"
Sep 19 08:55:01.703474 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.703391    2584 scope.go:117] "RemoveContainer" containerID="12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"
Sep 19 08:55:01.713441 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.713423    2584 scope.go:117] "RemoveContainer" containerID="80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"
Sep 19 08:55:01.722536 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.722506    2584 scope.go:117] "RemoveContainer" containerID="0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"
Sep 19 08:55:01.726915 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.726884    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:55:01.731702 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.731667    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-655bcc46cd-7cr2v"]
Sep 19 08:55:01.732429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.732409    2584 scope.go:117] "RemoveContainer" containerID="12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"
Sep 19 08:55:01.732788 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:55:01.732769    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc\": container with ID starting with 12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc not found: ID does not exist" containerID="12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"
Sep 19 08:55:01.732842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.732795    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc"} err="failed to get container status \"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc\": rpc error: code = NotFound desc = could not find container \"12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc\": container with ID starting with 12a964c0d63c0925f81627cda4ffb4f6e33519befc4c7d344bdcb31f9185d9bc not found: ID does not exist"
Sep 19 08:55:01.732842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.732811    2584 scope.go:117] "RemoveContainer" containerID="80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"
Sep 19 08:55:01.733104 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:55:01.733087    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af\": container with ID starting with 80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af not found: ID does not exist" containerID="80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"
Sep 19 08:55:01.733164 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.733109    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af"} err="failed to get container status \"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af\": rpc error: code = NotFound desc = could not find container \"80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af\": container with ID starting with 80d89eb114fb287c92125ed57000d45c2172c2974c6636336e269976349884af not found: ID does not exist"
Sep 19 08:55:01.733164 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.733123    2584 scope.go:117] "RemoveContainer" containerID="0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"
Sep 19 08:55:01.733411 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:55:01.733389    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba\": container with ID starting with 0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba not found: ID does not exist" containerID="0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"
Sep 19 08:55:01.733465 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.733419    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba"} err="failed to get container status \"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba\": rpc error: code = NotFound desc = could not find container \"0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba\": container with ID starting with 0341b2d72166fed514940abd9ca025ac2398800c63bf8798f383b6b670a3edba not found: ID does not exist"
Sep 19 08:55:01.914962 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:01.914916    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" path="/var/lib/kubelet/pods/0fe2812f-3572-4040-8776-9b8b6bb41a7c/volumes"
Sep 19 08:55:34.061758 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:34.061731    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:55:34.065184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:34.065163    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 08:55:34.066904 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:34.066883    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:55:34.070310 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:55:34.070287    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 08:56:19.587246 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:19.587174    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:56:19.587720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:19.587410    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" containerID="cri-o://55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385" gracePeriod=30
Sep 19 08:56:19.587720 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:19.587434    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kube-rbac-proxy" containerID="cri-o://3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f" gracePeriod=30
Sep 19 08:56:20.093446 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:20.093415    2584 generic.go:356] "Generic (PLEG): container finished" podID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerID="3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f" exitCode=2
Sep 19 08:56:20.093615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:20.093452    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerDied","Data":"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"}
Sep 19 08:56:22.209579 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:22.209537    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 19 08:56:22.213888 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:22.213862    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 19 08:56:24.027750 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.027724    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:56:24.113786 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.113752    2584 generic.go:356] "Generic (PLEG): container finished" podID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerID="55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385" exitCode=0
Sep 19 08:56:24.113963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.113808    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"
Sep 19 08:56:24.113963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.113822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerDied","Data":"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"}
Sep 19 08:56:24.113963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.113883    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh" event={"ID":"1106ad51-89e3-430f-b6f0-b65ed6175d5f","Type":"ContainerDied","Data":"e186f5599b4c4bd63a237517f7ab9abc1f3ce0f9a7071c987b5003609a280913"}
Sep 19 08:56:24.113963 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.113916    2584 scope.go:117] "RemoveContainer" containerID="3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"
Sep 19 08:56:24.122230 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.122213    2584 scope.go:117] "RemoveContainer" containerID="55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"
Sep 19 08:56:24.129546 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.129529    2584 scope.go:117] "RemoveContainer" containerID="1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"
Sep 19 08:56:24.136943 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.136929    2584 scope.go:117] "RemoveContainer" containerID="3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"
Sep 19 08:56:24.137216 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:56:24.137198    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f\": container with ID starting with 3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f not found: ID does not exist" containerID="3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"
Sep 19 08:56:24.137270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.137225    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f"} err="failed to get container status \"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f\": rpc error: code = NotFound desc = could not find container \"3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f\": container with ID starting with 3a21062a196ee4af2a463030f96f6869306f26a7082849392748a0d1ede73b9f not found: ID does not exist"
Sep 19 08:56:24.137270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.137243    2584 scope.go:117] "RemoveContainer" containerID="55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"
Sep 19 08:56:24.137477 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:56:24.137462    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385\": container with ID starting with 55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385 not found: ID does not exist" containerID="55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"
Sep 19 08:56:24.137516 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.137480    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385"} err="failed to get container status \"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385\": rpc error: code = NotFound desc = could not find container \"55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385\": container with ID starting with 55511d8a6307808d2c037ef667d7807ec00206e96e6566487273ad1925196385 not found: ID does not exist"
Sep 19 08:56:24.137516 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.137492    2584 scope.go:117] "RemoveContainer" containerID="1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"
Sep 19 08:56:24.137731 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:56:24.137714    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0\": container with ID starting with 1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0 not found: ID does not exist" containerID="1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"
Sep 19 08:56:24.137788 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.137739    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0"} err="failed to get container status \"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0\": rpc error: code = NotFound desc = could not find container \"1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0\": container with ID starting with 1ad0d5e55b3d15807717b4102b40d3523e485633099a0b143cec59b61ed356d0 not found: ID does not exist"
Sep 19 08:56:24.151032 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151013    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location\") pod \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") "
Sep 19 08:56:24.151102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151051    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") pod \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") "
Sep 19 08:56:24.151102 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151085    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert\") pod \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") "
Sep 19 08:56:24.151216 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151192    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkvrz\" (UniqueName: \"kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz\") pod \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") "
Sep 19 08:56:24.151270 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151259    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config\") pod \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\" (UID: \"1106ad51-89e3-430f-b6f0-b65ed6175d5f\") "
Sep 19 08:56:24.151381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151359    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1106ad51-89e3-430f-b6f0-b65ed6175d5f" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:24.151434 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1106ad51-89e3-430f-b6f0-b65ed6175d5f" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:24.151580 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151563    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.151629 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151574    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1106ad51-89e3-430f-b6f0-b65ed6175d5f" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:24.151629 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.151586    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.152930 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.152910    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1106ad51-89e3-430f-b6f0-b65ed6175d5f" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:56:24.153115 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.153095    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz" (OuterVolumeSpecName: "kube-api-access-jkvrz") pod "1106ad51-89e3-430f-b6f0-b65ed6175d5f" (UID: "1106ad51-89e3-430f-b6f0-b65ed6175d5f"). InnerVolumeSpecName "kube-api-access-jkvrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:24.252774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.252743    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvrz\" (UniqueName: \"kubernetes.io/projected/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-api-access-jkvrz\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.252774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.252771    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1106ad51-89e3-430f-b6f0-b65ed6175d5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.252774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.252781    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1106ad51-89e3-430f-b6f0-b65ed6175d5f-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:56:24.441816 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.441785    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:56:24.443668 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.443640    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-78pfh"]
Sep 19 08:56:24.806846 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.806814    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:56:24.807406 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807390    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807409    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807422    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807431    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807441    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807450    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container"
Sep 19 08:56:24.807468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807460    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807470    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807480    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="storage-initializer"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807488    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="storage-initializer"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807502    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807510    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807521    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="storage-initializer"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807529    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="storage-initializer"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807546    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807554    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kube-rbac-proxy"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807572    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="storage-initializer"
Sep 19 08:56:24.807650 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807580    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="storage-initializer"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kube-rbac-proxy"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807670    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kserve-container"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807680    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" containerName="kserve-container"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807689    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kube-rbac-proxy"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807699    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe2812f-3572-4040-8776-9b8b6bb41a7c" containerName="kube-rbac-proxy"
Sep 19 08:56:24.808002 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.807707    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="500f3b73-e44f-4854-bc5e-ca13bcdc70f1" containerName="kserve-container"
Sep 19 08:56:24.812956 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.812934    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:24.817072 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 19 08:56:24.817199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817100    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:56:24.817199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817100    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:56:24.817199 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817134    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 19 08:56:24.817374 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817255    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:56:24.817493 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817476    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:56:24.817569 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.817521    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:56:24.822991 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.822951    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:56:24.827752 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.827719    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.828410 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.828348    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:56:24.831045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.831024    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 19 08:56:24.836828 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.836456    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:56:24.957803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.957782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg7j\" (UniqueName: \"kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.957936 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.957813    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:24.957936 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.957849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.957936 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.957922    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.958139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.957960    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:24.958139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.958011    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.958139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.958038    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svz2d\" (UniqueName: \"kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:24.958139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.958066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:24.958139 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.958118    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:24.958348 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:24.958193    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059191 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg7j\" (UniqueName: \"kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059255    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059372    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svz2d\" (UniqueName: \"kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:56:25.059536    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:56:25.059624 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:56:25.059592    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls podName:08b74540-d700-4fdc-8148-771cdf6b554e nodeName:}" failed. No retries permitted until 2026-09-19 08:56:25.559571838 +0000 UTC m=+2153.644072623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-wp85q" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:56:25.060207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059684    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.060207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.059951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.060207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.060098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.060207 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.060163    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.060397 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.060379    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.060497 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.060480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.061977 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.061951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.067396 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.067368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svz2d\" (UniqueName: \"kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.067652 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.067633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg7j\" (UniqueName: \"kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.125978 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.125955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:25.399109 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.399040    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:56:25.402304 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:56:25.402278    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bfe51b_2e0a_4e4c_84ad_cc4b73c6a7a2.slice/crio-cffdf0e72aa70425f64d915ea6d30977d14ee31644e408979c270a8663e8c8ef WatchSource:0}: Error finding container cffdf0e72aa70425f64d915ea6d30977d14ee31644e408979c270a8663e8c8ef: Status 404 returned error can't find the container with id cffdf0e72aa70425f64d915ea6d30977d14ee31644e408979c270a8663e8c8ef
Sep 19 08:56:25.564605 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.564574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.566839 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.566816    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wp85q\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.741571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.741540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:25.907148 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:25.907115    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1106ad51-89e3-430f-b6f0-b65ed6175d5f" path="/var/lib/kubelet/pods/1106ad51-89e3-430f-b6f0-b65ed6175d5f/volumes"
Sep 19 08:56:26.038185 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:56:26.038158    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b74540_d700_4fdc_8148_771cdf6b554e.slice/crio-2036251b5579d9cde3b6103f6dedba0d2effbeca72c3d0ed194a0724d1f77b24 WatchSource:0}: Error finding container 2036251b5579d9cde3b6103f6dedba0d2effbeca72c3d0ed194a0724d1f77b24: Status 404 returned error can't find the container with id 2036251b5579d9cde3b6103f6dedba0d2effbeca72c3d0ed194a0724d1f77b24
Sep 19 08:56:26.038315 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:26.038249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:56:26.125564 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:26.125530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerStarted","Data":"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"}
Sep 19 08:56:26.125898 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:26.125589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerStarted","Data":"cffdf0e72aa70425f64d915ea6d30977d14ee31644e408979c270a8663e8c8ef"}
Sep 19 08:56:26.127256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:26.127232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerStarted","Data":"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"}
Sep 19 08:56:26.127357 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:26.127285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerStarted","Data":"2036251b5579d9cde3b6103f6dedba0d2effbeca72c3d0ed194a0724d1f77b24"}
Sep 19 08:56:29.143862 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:29.143833    2584 generic.go:356] "Generic (PLEG): container finished" podID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerID="59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56" exitCode=0
Sep 19 08:56:29.144280 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:29.143891    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerDied","Data":"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"}
Sep 19 08:56:30.150231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.150149    2584 generic.go:356] "Generic (PLEG): container finished" podID="08b74540-d700-4fdc-8148-771cdf6b554e" containerID="276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350" exitCode=0
Sep 19 08:56:30.150231 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.150220    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerDied","Data":"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"}
Sep 19 08:56:30.152561 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.152538    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerStarted","Data":"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"}
Sep 19 08:56:30.152672 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.152596    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerStarted","Data":"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"}
Sep 19 08:56:30.152812 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.152797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:30.187313 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:30.187271    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podStartSLOduration=2.524846221 podStartE2EDuration="6.187257248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662411027s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:30.186261065 +0000 UTC m=+2158.270761870" watchObservedRunningTime="2026-09-19 08:56:30.187257248 +0000 UTC m=+2158.271758050"
Sep 19 08:56:31.160243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.160209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerStarted","Data":"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"}
Sep 19 08:56:31.160676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.160268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerStarted","Data":"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"}
Sep 19 08:56:31.160676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.160430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:31.160676 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.160629    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:31.162043 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.162017    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:56:31.180468 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:31.180416    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podStartSLOduration=3.560650646 podStartE2EDuration="7.180404186s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61975354s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:31.178216212 +0000 UTC m=+2159.262717014" watchObservedRunningTime="2026-09-19 08:56:31.180404186 +0000 UTC m=+2159.264904988"
Sep 19 08:56:32.170041 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:32.170010    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:32.170446 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:32.170136    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:56:32.171192 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:32.171163    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:56:33.173510 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:33.173465    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:56:37.174311 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:37.174281    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:56:37.174885 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:37.174859    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:56:38.178373 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:38.178346    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:56:38.178881 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:38.178852    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:56:47.175684 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:47.175641    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:56:48.178863 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:48.178826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:56:57.175732 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:57.175695    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:56:58.179807 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:56:58.179770    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:57:07.175421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:07.175380    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:57:08.179152 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:08.179113    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:57:12.347964 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.347919    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:57:12.351927 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.351906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.354259 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.354234    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 19 08:57:12.354379 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.354239    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 19 08:57:12.360544 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.360262    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:57:12.446422 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.446377    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.446601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.446438    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.446601 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.446514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.446683 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.446641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zq7v\" (UniqueName: \"kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.446729 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.446714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.547576 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.547537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zq7v\" (UniqueName: \"kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.547765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.547614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.547765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.547642    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.547765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.547689    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.547765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.547732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.548015 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:12.547855    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 19 08:57:12.548015 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:12.547946    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls podName:1fe3929b-fb7b-46b4-94b5-57cd100dfc31 nodeName:}" failed. No retries permitted until 2026-09-19 08:57:13.047901857 +0000 UTC m=+2201.132402643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 19 08:57:12.548113 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.548082    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.548419 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.548403    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.548459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.548418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:12.557968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:12.557948    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zq7v\" (UniqueName: \"kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:13.052177 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:13.052145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:13.054435 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:13.054411    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:13.264261 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:13.264228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:13.543770 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:13.543745    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:57:13.545466 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:57:13.545437    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe3929b_fb7b_46b4_94b5_57cd100dfc31.slice/crio-45801195cfc22cfb2901cd2678fe716da3d9d446e42943e45cf8cabfe5296214 WatchSource:0}: Error finding container 45801195cfc22cfb2901cd2678fe716da3d9d446e42943e45cf8cabfe5296214: Status 404 returned error can't find the container with id 45801195cfc22cfb2901cd2678fe716da3d9d446e42943e45cf8cabfe5296214
Sep 19 08:57:14.376945 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:14.376908    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerStarted","Data":"4f1ea965bda7cd3b8c535c80ca70a54bb3874e877450796af6d7c0ed589573f5"}
Sep 19 08:57:14.377167 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:14.376976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerStarted","Data":"45801195cfc22cfb2901cd2678fe716da3d9d446e42943e45cf8cabfe5296214"}
Sep 19 08:57:17.174962 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:17.174928    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:57:18.178960 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:18.178923    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:57:18.399101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:18.399059    2584 generic.go:356] "Generic (PLEG): container finished" podID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerID="4f1ea965bda7cd3b8c535c80ca70a54bb3874e877450796af6d7c0ed589573f5" exitCode=0
Sep 19 08:57:18.399261 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:18.399133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerDied","Data":"4f1ea965bda7cd3b8c535c80ca70a54bb3874e877450796af6d7c0ed589573f5"}
Sep 19 08:57:18.400333 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:18.400312    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:57:22.431854 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:22.431822    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerStarted","Data":"e5368f38c19e4453dc0f91a8488062c70a1864cbd3152042466de84c645d2545"}
Sep 19 08:57:22.432295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:22.431886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerStarted","Data":"8e826bd0794faa098e4f48f9fef6bdb5a4d58137db9706a8c76296e051fb964c"}
Sep 19 08:57:22.432295 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:22.432131    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:22.450165 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:22.450111    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podStartSLOduration=2.643248045 podStartE2EDuration="10.450095605s" totalImagesPullingTime="3.46934691s" totalInitContainerRuntime="4.33750065s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:57:22.449153832 +0000 UTC m=+2210.533654634" watchObservedRunningTime="2026-09-19 08:57:22.450095605 +0000 UTC m=+2210.534596407"
Sep 19 08:57:23.437571 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:23.437542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:23.439004 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:23.438965    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 19 08:57:24.442637 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:24.442598    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 19 08:57:27.174885 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:27.174850    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:57:28.179764 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:28.179711    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:57:29.447150 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:29.447122    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:29.447582 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:29.447556    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 19 08:57:37.175476 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:37.175437    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:57:38.179338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:38.179295    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 19 08:57:39.447715 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:39.447686    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:57:47.176215 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.176144    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:57:47.299296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.299264    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:57:47.299543 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.299500    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container" containerID="cri-o://5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96" gracePeriod=30
Sep 19 08:57:47.299614 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.299567    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy" containerID="cri-o://4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14" gracePeriod=30
Sep 19 08:57:47.300315 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.300128    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:57:47.566036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.566007    2584 generic.go:356] "Generic (PLEG): container finished" podID="08b74540-d700-4fdc-8148-771cdf6b554e" containerID="4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14" exitCode=2
Sep 19 08:57:47.566170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:47.566031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerDied","Data":"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"}
Sep 19 08:57:48.174008 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:48.173950    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 19 08:57:51.354617 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:51.354584    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:57:51.355003 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:51.354851    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container" containerID="cri-o://8e826bd0794faa098e4f48f9fef6bdb5a4d58137db9706a8c76296e051fb964c" gracePeriod=30
Sep 19 08:57:51.355086 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:51.354864    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" containerID="cri-o://e5368f38c19e4453dc0f91a8488062c70a1864cbd3152042466de84c645d2545" gracePeriod=30
Sep 19 08:57:51.588949 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:51.588921    2584 generic.go:356] "Generic (PLEG): container finished" podID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerID="e5368f38c19e4453dc0f91a8488062c70a1864cbd3152042466de84c645d2545" exitCode=2
Sep 19 08:57:51.589101 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:51.589023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerDied","Data":"e5368f38c19e4453dc0f91a8488062c70a1864cbd3152042466de84c645d2545"}
Sep 19 08:57:53.236258 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.236235    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:57:53.302351 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302328    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") pod \"08b74540-d700-4fdc-8148-771cdf6b554e\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") "
Sep 19 08:57:53.302485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302394    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location\") pod \"08b74540-d700-4fdc-8148-771cdf6b554e\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") "
Sep 19 08:57:53.302485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302437    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert\") pod \"08b74540-d700-4fdc-8148-771cdf6b554e\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") "
Sep 19 08:57:53.302485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302481    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgg7j\" (UniqueName: \"kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j\") pod \"08b74540-d700-4fdc-8148-771cdf6b554e\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") "
Sep 19 08:57:53.302647 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302505    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config\") pod \"08b74540-d700-4fdc-8148-771cdf6b554e\" (UID: \"08b74540-d700-4fdc-8148-771cdf6b554e\") "
Sep 19 08:57:53.302767 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302730    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08b74540-d700-4fdc-8148-771cdf6b554e" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:53.302933 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302832    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08b74540-d700-4fdc-8148-771cdf6b554e-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:53.302933 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302840    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08b74540-d700-4fdc-8148-771cdf6b554e" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:53.303029 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.302931    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08b74540-d700-4fdc-8148-771cdf6b554e" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:53.304522 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.304496    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08b74540-d700-4fdc-8148-771cdf6b554e" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:53.304627 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.304589    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j" (OuterVolumeSpecName: "kube-api-access-kgg7j") pod "08b74540-d700-4fdc-8148-771cdf6b554e" (UID: "08b74540-d700-4fdc-8148-771cdf6b554e"). InnerVolumeSpecName "kube-api-access-kgg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:53.403391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.403338    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgg7j\" (UniqueName: \"kubernetes.io/projected/08b74540-d700-4fdc-8148-771cdf6b554e-kube-api-access-kgg7j\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:53.403391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.403360    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:53.403391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.403370    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08b74540-d700-4fdc-8148-771cdf6b554e-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:53.403391 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.403379    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08b74540-d700-4fdc-8148-771cdf6b554e-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:53.602338 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.602305    2584 generic.go:356] "Generic (PLEG): container finished" podID="08b74540-d700-4fdc-8148-771cdf6b554e" containerID="5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96" exitCode=0
Sep 19 08:57:53.602459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.602367    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"
Sep 19 08:57:53.602459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.602367    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerDied","Data":"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"}
Sep 19 08:57:53.602530 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.602473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" event={"ID":"08b74540-d700-4fdc-8148-771cdf6b554e","Type":"ContainerDied","Data":"2036251b5579d9cde3b6103f6dedba0d2effbeca72c3d0ed194a0724d1f77b24"}
Sep 19 08:57:53.602530 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.602514    2584 scope.go:117] "RemoveContainer" containerID="4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"
Sep 19 08:57:53.612125 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.612106    2584 scope.go:117] "RemoveContainer" containerID="5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"
Sep 19 08:57:53.619775 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.619758    2584 scope.go:117] "RemoveContainer" containerID="276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"
Sep 19 08:57:53.624893 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.624868    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:57:53.627228 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.627212    2584 scope.go:117] "RemoveContainer" containerID="4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"
Sep 19 08:57:53.627483 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:53.627464    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14\": container with ID starting with 4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14 not found: ID does not exist" containerID="4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"
Sep 19 08:57:53.627538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.627492    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14"} err="failed to get container status \"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14\": rpc error: code = NotFound desc = could not find container \"4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14\": container with ID starting with 4f7fb7a4f4638056ea5083a3ccce37396931930ea5f9f1de6c27be8221dc4a14 not found: ID does not exist"
Sep 19 08:57:53.627538 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.627507    2584 scope.go:117] "RemoveContainer" containerID="5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"
Sep 19 08:57:53.627735 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:53.627717    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96\": container with ID starting with 5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96 not found: ID does not exist" containerID="5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"
Sep 19 08:57:53.627805 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.627746    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96"} err="failed to get container status \"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96\": rpc error: code = NotFound desc = could not find container \"5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96\": container with ID starting with 5b0d992f81bdbb8889c49e525574706f2e0198c272a8b6463ee5e11b07110d96 not found: ID does not exist"
Sep 19 08:57:53.627805 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.627766    2584 scope.go:117] "RemoveContainer" containerID="276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"
Sep 19 08:57:53.628073 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:53.628056    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350\": container with ID starting with 276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350 not found: ID does not exist" containerID="276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"
Sep 19 08:57:53.628127 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.628078    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350"} err="failed to get container status \"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350\": rpc error: code = NotFound desc = could not find container \"276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350\": container with ID starting with 276988424b796b73aa128ce8c89150d61af77d0925f94adabed42d55a812e350 not found: ID does not exist"
Sep 19 08:57:53.630917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.630898    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q"]
Sep 19 08:57:53.907634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:53.907602    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" path="/var/lib/kubelet/pods/08b74540-d700-4fdc-8148-771cdf6b554e/volumes"
Sep 19 08:57:54.174668 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.174593    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wp85q" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": context deadline exceeded"
Sep 19 08:57:54.443510 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.443441    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:57:54.798826 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.798796    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:57:54.799053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.799023    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" containerID="cri-o://502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20" gracePeriod=30
Sep 19 08:57:54.799195 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.799075    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kube-rbac-proxy" containerID="cri-o://9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f" gracePeriod=30
Sep 19 08:57:54.842071 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842045    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:57:54.842414 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842403    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy"
Sep 19 08:57:54.842456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842416    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy"
Sep 19 08:57:54.842456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842441    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="storage-initializer"
Sep 19 08:57:54.842456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842447    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="storage-initializer"
Sep 19 08:57:54.842456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842453    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container"
Sep 19 08:57:54.842574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842458    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container"
Sep 19 08:57:54.842574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842509    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kube-rbac-proxy"
Sep 19 08:57:54.842574 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.842518    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b74540-d700-4fdc-8148-771cdf6b554e" containerName="kserve-container"
Sep 19 08:57:54.844669 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.844654    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:54.846991 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.846963    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 19 08:57:54.855539 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.855519    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:57:54.914604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.914581    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brm24\" (UniqueName: \"kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:54.914735 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.914613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:54.914735 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.914692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:54.914828 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.914761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:54.914828 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:54.914777    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016046 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brm24\" (UniqueName: \"kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016153 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016522 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016504    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016704 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.016767 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.016735    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.018466 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.018450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.024235 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.024219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brm24\" (UniqueName: \"kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24\") pod \"isvc-canary-rollback-predictor-7f7b644c9-cwr62\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.154656 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.154602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:57:55.424876 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.424854    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:57:55.426681 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:57:55.426655    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8672a6_798a_49da_9f40_2fb67b40cf2e.slice/crio-35055b20c7af18ce6bafb9482f24eb12f4041d0f0f9f4e160d99e29f9c4890b2 WatchSource:0}: Error finding container 35055b20c7af18ce6bafb9482f24eb12f4041d0f0f9f4e160d99e29f9c4890b2: Status 404 returned error can't find the container with id 35055b20c7af18ce6bafb9482f24eb12f4041d0f0f9f4e160d99e29f9c4890b2
Sep 19 08:57:55.615958 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.615904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerStarted","Data":"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"}
Sep 19 08:57:55.616397 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.616010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerStarted","Data":"35055b20c7af18ce6bafb9482f24eb12f4041d0f0f9f4e160d99e29f9c4890b2"}
Sep 19 08:57:55.618223 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.618196    2584 generic.go:356] "Generic (PLEG): container finished" podID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerID="9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f" exitCode=2
Sep 19 08:57:55.618352 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:55.618273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerDied","Data":"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"}
Sep 19 08:57:57.169946 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:57.169905    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 19 08:57:57.174861 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:57.174834    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 19 08:57:59.048355 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.048333    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:57:59.150614 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.150558    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert\") pod \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") "
Sep 19 08:57:59.150737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.150637    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config\") pod \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") "
Sep 19 08:57:59.150737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.150682    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svz2d\" (UniqueName: \"kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d\") pod \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") "
Sep 19 08:57:59.150737 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.150709    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location\") pod \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") "
Sep 19 08:57:59.150900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.150782    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls\") pod \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\" (UID: \"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2\") "
Sep 19 08:57:59.151056 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.151024    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" (UID: "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:59.151056 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.151035    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" (UID: "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:57:59.151214 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.151034    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" (UID: "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:59.152639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.152619    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" (UID: "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:59.152701 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.152624    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d" (OuterVolumeSpecName: "kube-api-access-svz2d") pod "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" (UID: "92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2"). InnerVolumeSpecName "kube-api-access-svz2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:59.251521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.251496    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:59.251521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.251518    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:59.251659 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.251527    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:59.251659 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.251537    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svz2d\" (UniqueName: \"kubernetes.io/projected/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kube-api-access-svz2d\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:59.251659 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.251545    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:57:59.443181 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.443123    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:57:59.639068 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.639039    2584 generic.go:356] "Generic (PLEG): container finished" podID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerID="502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20" exitCode=0
Sep 19 08:57:59.639180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.639100    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"
Sep 19 08:57:59.639180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.639120    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerDied","Data":"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"}
Sep 19 08:57:59.639180 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.639172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5" event={"ID":"92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2","Type":"ContainerDied","Data":"cffdf0e72aa70425f64d915ea6d30977d14ee31644e408979c270a8663e8c8ef"}
Sep 19 08:57:59.639312 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.639199    2584 scope.go:117] "RemoveContainer" containerID="9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"
Sep 19 08:57:59.640881 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.640864    2584 generic.go:356] "Generic (PLEG): container finished" podID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerID="1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69" exitCode=0
Sep 19 08:57:59.641001 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.640894    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerDied","Data":"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"}
Sep 19 08:57:59.647968 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.647951    2584 scope.go:117] "RemoveContainer" containerID="502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"
Sep 19 08:57:59.656296 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.656278    2584 scope.go:117] "RemoveContainer" containerID="59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"
Sep 19 08:57:59.665912 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.665892    2584 scope.go:117] "RemoveContainer" containerID="9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"
Sep 19 08:57:59.666201 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:59.666180    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f\": container with ID starting with 9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f not found: ID does not exist" containerID="9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"
Sep 19 08:57:59.666254 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.666208    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f"} err="failed to get container status \"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f\": rpc error: code = NotFound desc = could not find container \"9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f\": container with ID starting with 9e134ebe4e37182292ea4e066bdf299e7f67ccb8fc404fd1143cfaef0b942f3f not found: ID does not exist"
Sep 19 08:57:59.666254 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.666224    2584 scope.go:117] "RemoveContainer" containerID="502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"
Sep 19 08:57:59.666477 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:59.666460    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20\": container with ID starting with 502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20 not found: ID does not exist" containerID="502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"
Sep 19 08:57:59.666517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.666485    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20"} err="failed to get container status \"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20\": rpc error: code = NotFound desc = could not find container \"502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20\": container with ID starting with 502a1469ff37b276478049a860abec973ed5245b15330e14d25a6c3d58401e20 not found: ID does not exist"
Sep 19 08:57:59.666517 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.666500    2584 scope.go:117] "RemoveContainer" containerID="59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"
Sep 19 08:57:59.666742 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:57:59.666725    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56\": container with ID starting with 59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56 not found: ID does not exist" containerID="59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"
Sep 19 08:57:59.666795 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.666745    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56"} err="failed to get container status \"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56\": rpc error: code = NotFound desc = could not find container \"59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56\": container with ID starting with 59ccef6cd2089d0c445d860c555b88d5ea947d3b85eccc8e13b685b883181e56 not found: ID does not exist"
Sep 19 08:57:59.672189 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.672170    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:57:59.676565 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.676545    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-dcns5"]
Sep 19 08:57:59.906995 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:57:59.906953    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" path="/var/lib/kubelet/pods/92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2/volumes"
Sep 19 08:58:00.647119 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.647088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerStarted","Data":"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"}
Sep 19 08:58:00.647521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.647159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerStarted","Data":"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"}
Sep 19 08:58:00.647521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.647471    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:58:00.647649 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.647622    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:58:00.648944 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.648919    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:00.667138 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:00.667098    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podStartSLOduration=3.062138867 podStartE2EDuration="6.667086414s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604947547s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:58:00.664320582 +0000 UTC m=+2248.748821383" watchObservedRunningTime="2026-09-19 08:58:00.667086414 +0000 UTC m=+2248.751587263"
Sep 19 08:58:01.653922 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:01.653878    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:04.443615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:04.443573    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:58:04.444008 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:04.443691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:58:06.657942 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:06.657911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:58:06.658497 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:06.658474    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:09.443156 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:09.443113    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:58:14.443521 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:14.443479    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:58:16.658842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:16.658801    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:19.443385 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:19.443344    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 19 08:58:21.750276 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:21.750243    2584 generic.go:356] "Generic (PLEG): container finished" podID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerID="8e826bd0794faa098e4f48f9fef6bdb5a4d58137db9706a8c76296e051fb964c" exitCode=137
Sep 19 08:58:21.750622 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:21.750312    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerDied","Data":"8e826bd0794faa098e4f48f9fef6bdb5a4d58137db9706a8c76296e051fb964c"}
Sep 19 08:58:21.998745 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:21.998717    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:58:22.127435 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127408    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") pod \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") "
Sep 19 08:58:22.127586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config\") pod \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") "
Sep 19 08:58:22.127586 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127472    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert\") pod \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") "
Sep 19 08:58:22.127680 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127584    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zq7v\" (UniqueName: \"kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v\") pod \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") "
Sep 19 08:58:22.127680 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127631    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location\") pod \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\" (UID: \"1fe3929b-fb7b-46b4-94b5-57cd100dfc31\") "
Sep 19 08:58:22.127817 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127796    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fe3929b-fb7b-46b4-94b5-57cd100dfc31" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:58:22.127817 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.127807    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1fe3929b-fb7b-46b4-94b5-57cd100dfc31" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:58:22.128141 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.128119    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:58:22.128256 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.128143    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:58:22.129604 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.129578    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fe3929b-fb7b-46b4-94b5-57cd100dfc31" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:58:22.129698 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.129653    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v" (OuterVolumeSpecName: "kube-api-access-8zq7v") pod "1fe3929b-fb7b-46b4-94b5-57cd100dfc31" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31"). InnerVolumeSpecName "kube-api-access-8zq7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:58:22.137459 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.137438    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fe3929b-fb7b-46b4-94b5-57cd100dfc31" (UID: "1fe3929b-fb7b-46b4-94b5-57cd100dfc31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:58:22.229429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.229400    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:58:22.229429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.229426    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zq7v\" (UniqueName: \"kubernetes.io/projected/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kube-api-access-8zq7v\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:58:22.229550 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.229440    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe3929b-fb7b-46b4-94b5-57cd100dfc31-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:58:22.757223 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.757186    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj" event={"ID":"1fe3929b-fb7b-46b4-94b5-57cd100dfc31","Type":"ContainerDied","Data":"45801195cfc22cfb2901cd2678fe716da3d9d446e42943e45cf8cabfe5296214"}
Sep 19 08:58:22.757616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.757239    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"
Sep 19 08:58:22.757616 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.757270    2584 scope.go:117] "RemoveContainer" containerID="e5368f38c19e4453dc0f91a8488062c70a1864cbd3152042466de84c645d2545"
Sep 19 08:58:22.765917 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.765901    2584 scope.go:117] "RemoveContainer" containerID="8e826bd0794faa098e4f48f9fef6bdb5a4d58137db9706a8c76296e051fb964c"
Sep 19 08:58:22.773657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.773643    2584 scope.go:117] "RemoveContainer" containerID="4f1ea965bda7cd3b8c535c80ca70a54bb3874e877450796af6d7c0ed589573f5"
Sep 19 08:58:22.779665 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.779644    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:58:22.786316 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:22.786295    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-pc7fj"]
Sep 19 08:58:23.906909 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:23.906876    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" path="/var/lib/kubelet/pods/1fe3929b-fb7b-46b4-94b5-57cd100dfc31/volumes"
Sep 19 08:58:26.659225 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:26.659186    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:36.659394 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:36.659352    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:46.658476 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:46.658439    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:58:56.658429 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:58:56.658388    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:59:06.659373 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:06.659335    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:59:16.659294 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:16.659204    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:59:24.921082 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:24.921048    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:59:24.921568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:24.921289    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" containerID="cri-o://c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887" gracePeriod=30
Sep 19 08:59:24.921568 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:24.921331    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kube-rbac-proxy" containerID="cri-o://0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786" gracePeriod=30
Sep 19 08:59:25.052282 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:25.052255    2584 generic.go:356] "Generic (PLEG): container finished" podID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerID="0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786" exitCode=2
Sep 19 08:59:25.052377 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:25.052329    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerDied","Data":"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"}
Sep 19 08:59:26.654705 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:26.654662    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 19 08:59:26.659053 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:26.659027    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 19 08:59:28.957042 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:28.957021    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:59:29.049734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.049675    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert\") pod \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") "
Sep 19 08:59:29.049734 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.049730    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brm24\" (UniqueName: \"kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24\") pod \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") "
Sep 19 08:59:29.049921 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.049773    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location\") pod \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") "
Sep 19 08:59:29.049921 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.049790    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config\") pod \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") "
Sep 19 08:59:29.049921 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.049843    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls\") pod \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\" (UID: \"6f8672a6-798a-49da-9f40-2fb67b40cf2e\") "
Sep 19 08:59:29.050103 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050070    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f8672a6-798a-49da-9f40-2fb67b40cf2e" (UID: "6f8672a6-798a-49da-9f40-2fb67b40cf2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:59:29.050240 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050133    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f8672a6-798a-49da-9f40-2fb67b40cf2e" (UID: "6f8672a6-798a-49da-9f40-2fb67b40cf2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:29.050382 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050360    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f8672a6-798a-49da-9f40-2fb67b40cf2e" (UID: "6f8672a6-798a-49da-9f40-2fb67b40cf2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:59:29.050515 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050493    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:59:29.050579 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050525    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:59:29.050579 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.050541    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:59:29.051946 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.051918    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24" (OuterVolumeSpecName: "kube-api-access-brm24") pod "6f8672a6-798a-49da-9f40-2fb67b40cf2e" (UID: "6f8672a6-798a-49da-9f40-2fb67b40cf2e"). InnerVolumeSpecName "kube-api-access-brm24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:59:29.052056 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.051953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f8672a6-798a-49da-9f40-2fb67b40cf2e" (UID: "6f8672a6-798a-49da-9f40-2fb67b40cf2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:59:29.071792 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.071764    2584 generic.go:356] "Generic (PLEG): container finished" podID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerID="c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887" exitCode=0
Sep 19 08:59:29.071900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.071830    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"
Sep 19 08:59:29.071900 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.071840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerDied","Data":"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"}
Sep 19 08:59:29.072045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.071902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62" event={"ID":"6f8672a6-798a-49da-9f40-2fb67b40cf2e","Type":"ContainerDied","Data":"35055b20c7af18ce6bafb9482f24eb12f4041d0f0f9f4e160d99e29f9c4890b2"}
Sep 19 08:59:29.072045 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.071931    2584 scope.go:117] "RemoveContainer" containerID="0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"
Sep 19 08:59:29.080355 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.080341    2584 scope.go:117] "RemoveContainer" containerID="c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"
Sep 19 08:59:29.087637 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.087622    2584 scope.go:117] "RemoveContainer" containerID="1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"
Sep 19 08:59:29.095703 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.095297    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:59:29.098125 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.098103    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-cwr62"]
Sep 19 08:59:29.098460 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.098445    2584 scope.go:117] "RemoveContainer" containerID="0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"
Sep 19 08:59:29.098742 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:59:29.098724    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786\": container with ID starting with 0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786 not found: ID does not exist" containerID="0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"
Sep 19 08:59:29.098792 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.098749    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786"} err="failed to get container status \"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786\": rpc error: code = NotFound desc = could not find container \"0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786\": container with ID starting with 0f94c05fd4590c0f4363ed9abd8ddf96f25707c592e83829dc68e4051e2e2786 not found: ID does not exist"
Sep 19 08:59:29.098792 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.098765    2584 scope.go:117] "RemoveContainer" containerID="c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"
Sep 19 08:59:29.099104 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:59:29.099086    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887\": container with ID starting with c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887 not found: ID does not exist" containerID="c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"
Sep 19 08:59:29.099170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.099110    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887"} err="failed to get container status \"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887\": rpc error: code = NotFound desc = could not find container \"c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887\": container with ID starting with c73a620db8686e35d085fd1d76b7d28e4933141efdfa8b56049a5862e667a887 not found: ID does not exist"
Sep 19 08:59:29.099170 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.099125    2584 scope.go:117] "RemoveContainer" containerID="1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"
Sep 19 08:59:29.099404 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:59:29.099376    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69\": container with ID starting with 1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69 not found: ID does not exist" containerID="1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"
Sep 19 08:59:29.099461 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.099402    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69"} err="failed to get container status \"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69\": rpc error: code = NotFound desc = could not find container \"1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69\": container with ID starting with 1723e7fd9ceb3269cb9e5e15caf7a158b19c56b08eeb96b90ae0bfc360951c69 not found: ID does not exist"
Sep 19 08:59:29.151197 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.151174    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8672a6-798a-49da-9f40-2fb67b40cf2e-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:59:29.151277 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.151197    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brm24\" (UniqueName: \"kubernetes.io/projected/6f8672a6-798a-49da-9f40-2fb67b40cf2e-kube-api-access-brm24\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 08:59:29.907109 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:29.907079    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" path="/var/lib/kubelet/pods/6f8672a6-798a-49da-9f40-2fb67b40cf2e/volumes"
Sep 19 08:59:30.153475 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153437    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153822    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="storage-initializer"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153834    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="storage-initializer"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153842    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153849    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153861    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153868    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153880    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153885    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153896    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kube-rbac-proxy"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153901    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kube-rbac-proxy"
Sep 19 08:59:30.153903 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153910    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="storage-initializer"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153915    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="storage-initializer"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153922    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153927    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153932    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kube-rbac-proxy"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.153937    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kube-rbac-proxy"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154004    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="storage-initializer"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154013    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="storage-initializer"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154141    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kube-rbac-proxy"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154164    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kube-rbac-proxy"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154178    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bfe51b-2e0a-4e4c-84ad-cc4b73c6a7a2" containerName="kserve-container"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154189    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kube-rbac-proxy"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154198    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8672a6-798a-49da-9f40-2fb67b40cf2e" containerName="kserve-container"
Sep 19 08:59:30.154999 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.154207    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe3929b-fb7b-46b4-94b5-57cd100dfc31" containerName="kserve-container"
Sep 19 08:59:30.157634 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.157589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.157711 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.157651    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 08:59:30.160227 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.160204    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:59:30.160227 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.160204    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 19 08:59:30.160415 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.160248    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:59:30.161184 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.161167    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 19 08:59:30.161279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.161218    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 08:59:30.161279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.161242    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:59:30.161279 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.161256    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:59:30.161639 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.161625    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.163803 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.163779    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 19 08:59:30.170485 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.170464    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 08:59:30.173036 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.173015    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 08:59:30.259132 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxm25\" (UniqueName: \"kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.259249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.259249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.259249 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.259421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.259421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259311    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.259421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.259421 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.259615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259442    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcqb\" (UniqueName: \"kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.259615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.259479    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360282 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.360378 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcqb\" (UniqueName: \"kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360378 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxm25\" (UniqueName: \"kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.360456 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.360561 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360561 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360561 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.360561 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.360756 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.360585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.360756 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:59:30.360666    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 19 08:59:30.360756 ip-10-0-136-99 kubenswrapper[2584]: E0919 08:59:30.360746    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls podName:b71bdfac-0933-4392-bb47-5255bbd69dfa nodeName:}" failed. No retries permitted until 2026-09-19 08:59:30.860726037 +0000 UTC m=+2338.945226831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls") pod "isvc-canary-stop-predictor-5ffdccb94c-gcsfq" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361135    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361238    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361245    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.361281 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.361279    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.363339 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.363321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.368523 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.368495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcqb\" (UniqueName: \"kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.368615 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.368562    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxm25\" (UniqueName: \"kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-jbg89\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.471419 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.471395    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:30.747411 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.747381    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 08:59:30.749387 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:59:30.749360    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0153d3_8b06_49e4_a57d_dcf4c6a5366e.slice/crio-5cc8584cc98e26b01f45aebb22b944e74acb6e18591a4f56f5b9abfacef1229f WatchSource:0}: Error finding container 5cc8584cc98e26b01f45aebb22b944e74acb6e18591a4f56f5b9abfacef1229f: Status 404 returned error can't find the container with id 5cc8584cc98e26b01f45aebb22b944e74acb6e18591a4f56f5b9abfacef1229f
Sep 19 08:59:30.864792 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.864766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:30.866815 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:30.866795    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-gcsfq\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:31.079243 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:31.079163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:31.086765 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:31.086730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerStarted","Data":"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"}
Sep 19 08:59:31.086890 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:31.086791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerStarted","Data":"5cc8584cc98e26b01f45aebb22b944e74acb6e18591a4f56f5b9abfacef1229f"}
Sep 19 08:59:31.350288 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:31.350267    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 08:59:31.351915 ip-10-0-136-99 kubenswrapper[2584]: W0919 08:59:31.351885    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71bdfac_0933_4392_bb47_5255bbd69dfa.slice/crio-0bf70779fee32faf90a7b1cf2400d07e4ad726082ddc8cd6209b0bb4c3a81a32 WatchSource:0}: Error finding container 0bf70779fee32faf90a7b1cf2400d07e4ad726082ddc8cd6209b0bb4c3a81a32: Status 404 returned error can't find the container with id 0bf70779fee32faf90a7b1cf2400d07e4ad726082ddc8cd6209b0bb4c3a81a32
Sep 19 08:59:32.092381 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:32.092345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerStarted","Data":"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"}
Sep 19 08:59:32.092562 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:32.092405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerStarted","Data":"0bf70779fee32faf90a7b1cf2400d07e4ad726082ddc8cd6209b0bb4c3a81a32"}
Sep 19 08:59:35.107425 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:35.107393    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerID="1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40" exitCode=0
Sep 19 08:59:35.107774 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:35.107477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerDied","Data":"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"}
Sep 19 08:59:36.114114 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.114076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerStarted","Data":"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"}
Sep 19 08:59:36.114577 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.114148    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerStarted","Data":"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"}
Sep 19 08:59:36.114577 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.114378    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:36.115560 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.115538    2584 generic.go:356] "Generic (PLEG): container finished" podID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerID="5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593" exitCode=0
Sep 19 08:59:36.115657 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.115575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerDied","Data":"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"}
Sep 19 08:59:36.138642 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:36.138601    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podStartSLOduration=2.274227134 podStartE2EDuration="6.138587464s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86436033s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:59:36.138072168 +0000 UTC m=+2344.222572971" watchObservedRunningTime="2026-09-19 08:59:36.138587464 +0000 UTC m=+2344.223088266"
Sep 19 08:59:37.122971 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.122934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerStarted","Data":"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"}
Sep 19 08:59:37.123360 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.123030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerStarted","Data":"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"}
Sep 19 08:59:37.123360 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.123283    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:37.123504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.123423    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:37.123504 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.123477    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:37.124635 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.124611    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 08:59:37.124724 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.124630    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 08:59:37.146529 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:37.146494    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podStartSLOduration=3.325212676 podStartE2EDuration="7.146482743s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821270067s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:59:37.145742732 +0000 UTC m=+2345.230243533" watchObservedRunningTime="2026-09-19 08:59:37.146482743 +0000 UTC m=+2345.230983544"
Sep 19 08:59:38.127797 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:38.127760    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 08:59:38.128189 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:38.127882    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 08:59:43.132306 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:43.132271    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 08:59:43.132702 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:43.132583    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 08:59:43.132842 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:43.132804    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 08:59:43.133323 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:43.133297    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 08:59:53.133494 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:53.133458    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 08:59:53.133845 ip-10-0-136-99 kubenswrapper[2584]: I0919 08:59:53.133457    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:03.133780 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:03.133742    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:03.134241 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:03.133742    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:00:13.133412 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:13.133367    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:00:13.133813 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:13.133365    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:23.133035 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:23.132969    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:00:23.133424 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:23.133285    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:33.133212 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:33.133167    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:00:33.133590 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:33.133316    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:34.099396 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:34.099367    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:00:34.104915 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:34.104892    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:00:34.105592 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:34.105575    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:00:34.110334 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:34.110316    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:00:43.132964 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:43.132924    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:00:43.133373 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:43.133192    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:49.902031 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:49.901920    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 19 09:00:50.901999 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:50.901955    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 09:00:51.021344 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.021310    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 09:00:51.021726 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.021579    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container" containerID="cri-o://f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6" gracePeriod=30
Sep 19 09:00:51.021726 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.021621    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kube-rbac-proxy" containerID="cri-o://7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569" gracePeriod=30
Sep 19 09:00:51.022340 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.022158    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 09:00:51.484549 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.484513    2584 generic.go:356] "Generic (PLEG): container finished" podID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerID="7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569" exitCode=2
Sep 19 09:00:51.484755 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:51.484590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerDied","Data":"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"}
Sep 19 09:00:53.128936 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:53.128891    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 19 09:00:55.178077 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.178056    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 09:00:55.274895 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.274817    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlcqb\" (UniqueName: \"kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb\") pod \"b71bdfac-0933-4392-bb47-5255bbd69dfa\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") "
Sep 19 09:00:55.274895 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.274852    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert\") pod \"b71bdfac-0933-4392-bb47-5255bbd69dfa\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") "
Sep 19 09:00:55.274895 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.274874    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") pod \"b71bdfac-0933-4392-bb47-5255bbd69dfa\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") "
Sep 19 09:00:55.274895 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.274894    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config\") pod \"b71bdfac-0933-4392-bb47-5255bbd69dfa\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") "
Sep 19 09:00:55.275256 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.274930    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location\") pod \"b71bdfac-0933-4392-bb47-5255bbd69dfa\" (UID: \"b71bdfac-0933-4392-bb47-5255bbd69dfa\") "
Sep 19 09:00:55.275317 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.275250    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b71bdfac-0933-4392-bb47-5255bbd69dfa" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:00:55.275387 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.275265    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b71bdfac-0933-4392-bb47-5255bbd69dfa" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:55.275442 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.275414    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b71bdfac-0933-4392-bb47-5255bbd69dfa" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:00:55.276936 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.276903    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb" (OuterVolumeSpecName: "kube-api-access-dlcqb") pod "b71bdfac-0933-4392-bb47-5255bbd69dfa" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa"). InnerVolumeSpecName "kube-api-access-dlcqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:00:55.277056 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.276933    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b71bdfac-0933-4392-bb47-5255bbd69dfa" (UID: "b71bdfac-0933-4392-bb47-5255bbd69dfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:00:55.375870 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.375835    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:00:55.375870 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.375869    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b71bdfac-0933-4392-bb47-5255bbd69dfa-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:00:55.376049 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.375889    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:00:55.376049 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.375904    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b71bdfac-0933-4392-bb47-5255bbd69dfa-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:00:55.376049 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.375916    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcqb\" (UniqueName: \"kubernetes.io/projected/b71bdfac-0933-4392-bb47-5255bbd69dfa-kube-api-access-dlcqb\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:00:55.506636 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.506610    2584 generic.go:356] "Generic (PLEG): container finished" podID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerID="f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6" exitCode=0
Sep 19 09:00:55.506737 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.506645    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerDied","Data":"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"}
Sep 19 09:00:55.506737 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.506673    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"
Sep 19 09:00:55.506737 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.506722    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq" event={"ID":"b71bdfac-0933-4392-bb47-5255bbd69dfa","Type":"ContainerDied","Data":"0bf70779fee32faf90a7b1cf2400d07e4ad726082ddc8cd6209b0bb4c3a81a32"}
Sep 19 09:00:55.506846 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.506752    2584 scope.go:117] "RemoveContainer" containerID="7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"
Sep 19 09:00:55.516093 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.516061    2584 scope.go:117] "RemoveContainer" containerID="f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"
Sep 19 09:00:55.526392 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.526034    2584 scope.go:117] "RemoveContainer" containerID="5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"
Sep 19 09:00:55.528539 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.528519    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 09:00:55.531621 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.531600    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-gcsfq"]
Sep 19 09:00:55.534485 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.534469    2584 scope.go:117] "RemoveContainer" containerID="7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"
Sep 19 09:00:55.534733 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:00:55.534712    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569\": container with ID starting with 7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569 not found: ID does not exist" containerID="7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"
Sep 19 09:00:55.534794 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.534742    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569"} err="failed to get container status \"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569\": rpc error: code = NotFound desc = could not find container \"7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569\": container with ID starting with 7c473fe11cdc2e1a303e2c263ebf0fc31239127320f482549a53c622d38c6569 not found: ID does not exist"
Sep 19 09:00:55.534794 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.534762    2584 scope.go:117] "RemoveContainer" containerID="f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"
Sep 19 09:00:55.535039 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:00:55.535023    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6\": container with ID starting with f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6 not found: ID does not exist" containerID="f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"
Sep 19 09:00:55.535087 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.535045    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6"} err="failed to get container status \"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6\": rpc error: code = NotFound desc = could not find container \"f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6\": container with ID starting with f83c2c346b2c6cab1374a008621c1b9ad74edd8c986625f05fca2ccffc4b7dc6 not found: ID does not exist"
Sep 19 09:00:55.535087 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.535060    2584 scope.go:117] "RemoveContainer" containerID="5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"
Sep 19 09:00:55.535295 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:00:55.535280    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593\": container with ID starting with 5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593 not found: ID does not exist" containerID="5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"
Sep 19 09:00:55.535345 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.535298    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593"} err="failed to get container status \"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593\": rpc error: code = NotFound desc = could not find container \"5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593\": container with ID starting with 5ec61d09181e6cd5fcb79c4f56a22b20422f6eb8a8a28ce44991770845932593 not found: ID does not exist"
Sep 19 09:00:55.907958 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:00:55.907893    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" path="/var/lib/kubelet/pods/b71bdfac-0933-4392-bb47-5255bbd69dfa/volumes"
Sep 19 09:01:20.131187 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.131152    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 09:01:20.131682 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.131379    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" containerID="cri-o://ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306" gracePeriod=30
Sep 19 09:01:20.131682 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.131475    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kube-rbac-proxy" containerID="cri-o://4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a" gracePeriod=30
Sep 19 09:01:20.635777 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.635739    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerID="4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a" exitCode=2
Sep 19 09:01:20.635951 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.635812    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerDied","Data":"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"}
Sep 19 09:01:20.901550 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:20.901467    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 19 09:01:23.128872 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:23.128833    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 19 09:01:24.283267 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.283242    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 09:01:24.408272 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408213    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls\") pod \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") "
Sep 19 09:01:24.408272 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408252    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config\") pod \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") "
Sep 19 09:01:24.408433 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408273    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxm25\" (UniqueName: \"kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25\") pod \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") "
Sep 19 09:01:24.408433 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408306    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location\") pod \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") "
Sep 19 09:01:24.408433 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408336    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert\") pod \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\" (UID: \"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e\") "
Sep 19 09:01:24.408705 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408639    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" (UID: "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:01:24.408836 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408757    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" (UID: "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:24.408836 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.408804    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" (UID: "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:01:24.410418 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.410394    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" (UID: "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:01:24.410516 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.410499    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25" (OuterVolumeSpecName: "kube-api-access-gxm25") pod "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" (UID: "0f0153d3-8b06-49e4-a57d-dcf4c6a5366e"). InnerVolumeSpecName "kube-api-access-gxm25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:01:24.508998 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.508963    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:01:24.509096 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.509006    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:01:24.509096 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.509021    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxm25\" (UniqueName: \"kubernetes.io/projected/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kube-api-access-gxm25\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:01:24.509096 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.509035    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:01:24.509096 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.509045    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:01:24.655401 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.655371    2584 generic.go:356] "Generic (PLEG): container finished" podID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerID="ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306" exitCode=0
Sep 19 09:01:24.655527 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.655415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerDied","Data":"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"}
Sep 19 09:01:24.655527 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.655437    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"
Sep 19 09:01:24.655527 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.655478    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89" event={"ID":"0f0153d3-8b06-49e4-a57d-dcf4c6a5366e","Type":"ContainerDied","Data":"5cc8584cc98e26b01f45aebb22b944e74acb6e18591a4f56f5b9abfacef1229f"}
Sep 19 09:01:24.655527 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.655518    2584 scope.go:117] "RemoveContainer" containerID="4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"
Sep 19 09:01:24.664893 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.664869    2584 scope.go:117] "RemoveContainer" containerID="ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"
Sep 19 09:01:24.672913 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.672896    2584 scope.go:117] "RemoveContainer" containerID="1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"
Sep 19 09:01:24.677851 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.677826    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 09:01:24.681327 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.681305    2584 scope.go:117] "RemoveContainer" containerID="4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"
Sep 19 09:01:24.681734 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:01:24.681708    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a\": container with ID starting with 4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a not found: ID does not exist" containerID="4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"
Sep 19 09:01:24.681815 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.681746    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a"} err="failed to get container status \"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a\": rpc error: code = NotFound desc = could not find container \"4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a\": container with ID starting with 4690f31abc3728f7873213cf4e645a2dbe6837b15c0060d4ca4ef7e95513ae2a not found: ID does not exist"
Sep 19 09:01:24.681815 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.681769    2584 scope.go:117] "RemoveContainer" containerID="ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"
Sep 19 09:01:24.682119 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:01:24.682098    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306\": container with ID starting with ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306 not found: ID does not exist" containerID="ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"
Sep 19 09:01:24.682210 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.682123    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306"} err="failed to get container status \"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306\": rpc error: code = NotFound desc = could not find container \"ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306\": container with ID starting with ba07f7b1cb542fd7a0ddcf5fb256a7e102786fcf243e79fecc41d7c7c83cd306 not found: ID does not exist"
Sep 19 09:01:24.682210 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.682138    2584 scope.go:117] "RemoveContainer" containerID="1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"
Sep 19 09:01:24.682442 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:01:24.682424    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40\": container with ID starting with 1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40 not found: ID does not exist" containerID="1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"
Sep 19 09:01:24.682501 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.682451    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40"} err="failed to get container status \"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40\": rpc error: code = NotFound desc = could not find container \"1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40\": container with ID starting with 1654b597336389fe26c1b0cd3e568a1acc63b897e95c58fb6122451b0d8abf40 not found: ID does not exist"
Sep 19 09:01:24.682620 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:24.682602    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-jbg89"]
Sep 19 09:01:25.907318 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:01:25.907287    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" path="/var/lib/kubelet/pods/0f0153d3-8b06-49e4-a57d-dcf4c6a5366e/volumes"
Sep 19 09:03:26.703014 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.702966    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:03:26.711513 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711481    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="storage-initializer"
Sep 19 09:03:26.711513 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711507    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="storage-initializer"
Sep 19 09:03:26.711513 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711517    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711523    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711533    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711538    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711556    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="storage-initializer"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711561    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="storage-initializer"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711570    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711575    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711582    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711587    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711666    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711675    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0153d3-8b06-49e4-a57d-dcf4c6a5366e" containerName="kserve-container"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711682    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kube-rbac-proxy"
Sep 19 09:03:26.711785 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.711696    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71bdfac-0933-4392-bb47-5255bbd69dfa" containerName="kserve-container"
Sep 19 09:03:26.715084 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.715063    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.717736 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.717699    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:03:26.717863 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.717726    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:03:26.717921 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.717908    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 19 09:03:26.718054 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.718037    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:03:26.718118 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.718054    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:03:26.718118 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.718103    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:03:26.718331 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.718316    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 19 09:03:26.718637 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.718613    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:03:26.826749 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.826724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.826896 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.826763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nln\" (UniqueName: \"kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.826896 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.826803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.826896 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.826830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.826896 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.826858    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927540 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927659 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nln\" (UniqueName: \"kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927659 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927659 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927659 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.927868 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:03:26.927706    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 19 09:03:26.927868 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:03:26.927777    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls podName:594ec401-3a19-4843-89e6-dc71a7ea3508 nodeName:}" failed. No retries permitted until 2026-09-19 09:03:27.42775751 +0000 UTC m=+2575.512258294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 19 09:03:26.927967 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.927930    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.928315 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.928298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.928355 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.928325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:26.936250 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:26.936225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nln\" (UniqueName: \"kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:27.432782 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:27.432715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:27.436321 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:27.436276    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:27.627903 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:27.627836    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:27.919230 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:27.919203    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:03:27.921754 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:03:27.921725    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594ec401_3a19_4843_89e6_dc71a7ea3508.slice/crio-5cb31a830919643e92c8da0890c2fed582e052afb0e3f0c37b0a9a16bf1ac81e WatchSource:0}: Error finding container 5cb31a830919643e92c8da0890c2fed582e052afb0e3f0c37b0a9a16bf1ac81e: Status 404 returned error can't find the container with id 5cb31a830919643e92c8da0890c2fed582e052afb0e3f0c37b0a9a16bf1ac81e
Sep 19 09:03:28.263156 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:28.263118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerStarted","Data":"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"}
Sep 19 09:03:28.263341 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:28.263185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerStarted","Data":"5cb31a830919643e92c8da0890c2fed582e052afb0e3f0c37b0a9a16bf1ac81e"}
Sep 19 09:03:32.291041 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:32.290972    2584 generic.go:356] "Generic (PLEG): container finished" podID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerID="f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779" exitCode=0
Sep 19 09:03:32.291455 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:32.291048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerDied","Data":"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"}
Sep 19 09:03:32.292438 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:32.292418    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:03:39.335339 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:39.335297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerStarted","Data":"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"}
Sep 19 09:03:39.335770 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:39.335371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerStarted","Data":"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"}
Sep 19 09:03:39.335770 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:39.335585    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:39.355903 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:39.355828    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podStartSLOduration=2.671276932 podStartE2EDuration="13.355810112s" totalImagesPullingTime="6.821942845s" totalInitContainerRuntime="3.862590335s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:03:39.354293469 +0000 UTC m=+2587.438794272" watchObservedRunningTime="2026-09-19 09:03:39.355810112 +0000 UTC m=+2587.440310915"
Sep 19 09:03:40.340484 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:40.340448    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:40.341964 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:40.341937    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:03:41.345140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:41.345101    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:03:46.351353 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:46.351262    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:03:46.351961 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:46.351929    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:03:56.352077 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:03:56.352033    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:06.352030 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:06.351959    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:16.352236 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:16.352191    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:26.352265 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:26.352218    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:36.352950 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:36.352903    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:46.352773 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:46.352734    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:47.907712 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:47.907663    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:04:57.908158 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:04:57.908119    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:05:07.907867 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:07.907826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 19 09:05:17.909097 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:17.909014    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:05:27.989956 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:27.989920    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:05:27.990589 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:27.990257    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container" containerID="cri-o://ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e" gracePeriod=30
Sep 19 09:05:27.990589 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:27.990344    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kube-rbac-proxy" containerID="cri-o://1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1" gracePeriod=30
Sep 19 09:05:28.904570 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:28.904538    2584 generic.go:356] "Generic (PLEG): container finished" podID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerID="1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1" exitCode=2
Sep 19 09:05:28.904740 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:28.904597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerDied","Data":"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"}
Sep 19 09:05:31.345872 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:31.345823    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 19 09:05:33.638442 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.638412    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:05:33.726562 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.726522    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9nln\" (UniqueName: \"kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln\") pod \"594ec401-3a19-4843-89e6-dc71a7ea3508\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") "
Sep 19 09:05:33.726562 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.726560    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config\") pod \"594ec401-3a19-4843-89e6-dc71a7ea3508\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") "
Sep 19 09:05:33.726912 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.726605    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") pod \"594ec401-3a19-4843-89e6-dc71a7ea3508\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") "
Sep 19 09:05:33.726912 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.726669    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert\") pod \"594ec401-3a19-4843-89e6-dc71a7ea3508\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") "
Sep 19 09:05:33.726912 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.726778    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location\") pod \"594ec401-3a19-4843-89e6-dc71a7ea3508\" (UID: \"594ec401-3a19-4843-89e6-dc71a7ea3508\") "
Sep 19 09:05:33.727131 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.727093    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "594ec401-3a19-4843-89e6-dc71a7ea3508" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:05:33.727131 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.727110    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "594ec401-3a19-4843-89e6-dc71a7ea3508" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:05:33.727244 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.727199    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:05:33.727244 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.727203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "594ec401-3a19-4843-89e6-dc71a7ea3508" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:33.727244 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.727220    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:05:33.728736 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.728706    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "594ec401-3a19-4843-89e6-dc71a7ea3508" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:05:33.728850 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.728746    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln" (OuterVolumeSpecName: "kube-api-access-f9nln") pod "594ec401-3a19-4843-89e6-dc71a7ea3508" (UID: "594ec401-3a19-4843-89e6-dc71a7ea3508"). InnerVolumeSpecName "kube-api-access-f9nln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:05:33.827850 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.827767    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9nln\" (UniqueName: \"kubernetes.io/projected/594ec401-3a19-4843-89e6-dc71a7ea3508-kube-api-access-f9nln\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:05:33.827850 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.827794    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/594ec401-3a19-4843-89e6-dc71a7ea3508-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:05:33.827850 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.827806    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/594ec401-3a19-4843-89e6-dc71a7ea3508-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:05:33.928731 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.928701    2584 generic.go:356] "Generic (PLEG): container finished" podID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerID="ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e" exitCode=0
Sep 19 09:05:33.928882 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.928747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerDied","Data":"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"}
Sep 19 09:05:33.928882 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.928763    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"
Sep 19 09:05:33.928882 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.928795    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb" event={"ID":"594ec401-3a19-4843-89e6-dc71a7ea3508","Type":"ContainerDied","Data":"5cb31a830919643e92c8da0890c2fed582e052afb0e3f0c37b0a9a16bf1ac81e"}
Sep 19 09:05:33.928882 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.928824    2584 scope.go:117] "RemoveContainer" containerID="1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"
Sep 19 09:05:33.937273 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.937255    2584 scope.go:117] "RemoveContainer" containerID="ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"
Sep 19 09:05:33.945881 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.945858    2584 scope.go:117] "RemoveContainer" containerID="f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"
Sep 19 09:05:33.946679 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.946655    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:05:33.953023 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.952996    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9rmgb"]
Sep 19 09:05:33.956308 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.956285    2584 scope.go:117] "RemoveContainer" containerID="1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"
Sep 19 09:05:33.956929 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:05:33.956902    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1\": container with ID starting with 1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1 not found: ID does not exist" containerID="1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"
Sep 19 09:05:33.957124 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.956941    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1"} err="failed to get container status \"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1\": rpc error: code = NotFound desc = could not find container \"1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1\": container with ID starting with 1c47e384184cc53fbb357372a2e50efdb03b81b21a3019351a937f36c3814ea1 not found: ID does not exist"
Sep 19 09:05:33.957124 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.956966    2584 scope.go:117] "RemoveContainer" containerID="ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"
Sep 19 09:05:33.957302 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:05:33.957279    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e\": container with ID starting with ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e not found: ID does not exist" containerID="ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"
Sep 19 09:05:33.957362 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.957308    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e"} err="failed to get container status \"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e\": rpc error: code = NotFound desc = could not find container \"ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e\": container with ID starting with ea8db76b2ff33c155c4e15ca396fd348ad501bea963b7e9a79c4f29029c4962e not found: ID does not exist"
Sep 19 09:05:33.957362 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.957323    2584 scope.go:117] "RemoveContainer" containerID="f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"
Sep 19 09:05:33.957598 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:05:33.957577    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779\": container with ID starting with f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779 not found: ID does not exist" containerID="f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"
Sep 19 09:05:33.957690 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:33.957601    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779"} err="failed to get container status \"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779\": rpc error: code = NotFound desc = could not find container \"f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779\": container with ID starting with f26186ab811e0d84486aa0ea940921db9268a57f272ba5ead7915d190b420779 not found: ID does not exist"
Sep 19 09:05:34.131776 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:34.131697    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:05:34.137134 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:34.137111    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:05:34.137504 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:34.137481    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:05:34.143185 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:34.143170    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:05:35.908188 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:05:35.908153    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" path="/var/lib/kubelet/pods/594ec401-3a19-4843-89e6-dc71a7ea3508/volumes"
Sep 19 09:06:21.103516 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103485    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103848    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103859    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103868    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kube-rbac-proxy"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103874    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kube-rbac-proxy"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103882    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="storage-initializer"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103888    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="storage-initializer"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103955    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kserve-container"
Sep 19 09:06:21.103969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.103967    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="594ec401-3a19-4843-89e6-dc71a7ea3508" containerName="kube-rbac-proxy"
Sep 19 09:06:21.107190 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.107168    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.109688 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.109659    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 19 09:06:21.110100 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110079    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:06:21.110202 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110186    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:06:21.110269 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110186    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:06:21.110896 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110880    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:06:21.111010 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110916    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:06:21.111010 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.110921    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 19 09:06:21.121693 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.121639    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:06:21.225545 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.225508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.225728 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.225565    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jx4j\" (UniqueName: \"kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.225728 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.225603    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.225728 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.225649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.225728 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.225685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327139 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327097    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327352 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jx4j\" (UniqueName: \"kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327352 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327352 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327312    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327523 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327714 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327696    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327949 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.327949 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.327936    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.329796 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.329774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.336591 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.336561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jx4j\" (UniqueName: \"kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j\") pod \"isvc-xgboost-v2-predictor-fcc668bc-rwb8l\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.420047 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.419927    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:21.694885 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:21.694868    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:06:21.697336 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:06:21.697304    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf661a46c_ad1e_4763_b03f_f3f118ab93b0.slice/crio-1a009a479c7ae6b04f3c1bbaa93519f47467d980ae860c1ab4cf26dbdbc18671 WatchSource:0}: Error finding container 1a009a479c7ae6b04f3c1bbaa93519f47467d980ae860c1ab4cf26dbdbc18671: Status 404 returned error can't find the container with id 1a009a479c7ae6b04f3c1bbaa93519f47467d980ae860c1ab4cf26dbdbc18671
Sep 19 09:06:22.163720 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:22.163682    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerStarted","Data":"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"}
Sep 19 09:06:22.164136 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:22.163738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerStarted","Data":"1a009a479c7ae6b04f3c1bbaa93519f47467d980ae860c1ab4cf26dbdbc18671"}
Sep 19 09:06:26.183557 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:26.183521    2584 generic.go:356] "Generic (PLEG): container finished" podID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerID="c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a" exitCode=0
Sep 19 09:06:26.183946 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:26.183589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerDied","Data":"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"}
Sep 19 09:06:46.296912 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:46.296874    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerStarted","Data":"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"}
Sep 19 09:06:46.297325 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:46.296941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerStarted","Data":"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"}
Sep 19 09:06:46.297325 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:46.297138    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:46.317768 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:46.317716    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podStartSLOduration=1.8703690389999998 podStartE2EDuration="25.317701952s" totalImagesPullingTime="19.594699693s" totalInitContainerRuntime="3.85263322s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:46.315070855 +0000 UTC m=+2774.399571658" watchObservedRunningTime="2026-09-19 09:06:46.317701952 +0000 UTC m=+2774.402202754"
Sep 19 09:06:47.302434 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:47.302404    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:47.303917 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:47.303888    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:06:48.307121 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:48.307084    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:06:53.312653 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:53.312621    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:06:53.313623 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:06:53.313589    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:03.314185 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:03.314142    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:13.313650 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:13.313606    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:23.313508 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:23.313468    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:33.314080 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:33.314020    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:43.313498 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:43.313459    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:07:53.314021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:07:53.313969    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:08:01.800666 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:01.800630    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:08:01.801244 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:01.800974    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" containerID="cri-o://4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660" gracePeriod=30
Sep 19 09:08:01.801244 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:01.801064    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kube-rbac-proxy" containerID="cri-o://bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0" gracePeriod=30
Sep 19 09:08:02.678876 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:02.678840    2584 generic.go:356] "Generic (PLEG): container finished" podID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerID="bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0" exitCode=2
Sep 19 09:08:02.679074 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:02.678896    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerDied","Data":"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"}
Sep 19 09:08:03.308156 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:03.308119    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.63:8643/healthz\": dial tcp 10.132.0.63:8643: connect: connection refused"
Sep 19 09:08:03.314031 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:03.314005    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Sep 19 09:08:06.255212 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.255185    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:08:06.398266 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398173    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location\") pod \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") "
Sep 19 09:08:06.398266 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398214    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config\") pod \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") "
Sep 19 09:08:06.398266 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398245    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jx4j\" (UniqueName: \"kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j\") pod \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") "
Sep 19 09:08:06.398557 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398278    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert\") pod \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") "
Sep 19 09:08:06.398557 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398326    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls\") pod \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\" (UID: \"f661a46c-ad1e-4763-b03f-f3f118ab93b0\") "
Sep 19 09:08:06.398686 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398636    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f661a46c-ad1e-4763-b03f-f3f118ab93b0" (UID: "f661a46c-ad1e-4763-b03f-f3f118ab93b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:08:06.398686 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398640    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f661a46c-ad1e-4763-b03f-f3f118ab93b0" (UID: "f661a46c-ad1e-4763-b03f-f3f118ab93b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:06.398777 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.398697    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f661a46c-ad1e-4763-b03f-f3f118ab93b0" (UID: "f661a46c-ad1e-4763-b03f-f3f118ab93b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:08:06.400524 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.400502    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f661a46c-ad1e-4763-b03f-f3f118ab93b0" (UID: "f661a46c-ad1e-4763-b03f-f3f118ab93b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:08:06.400615 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.400521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j" (OuterVolumeSpecName: "kube-api-access-7jx4j") pod "f661a46c-ad1e-4763-b03f-f3f118ab93b0" (UID: "f661a46c-ad1e-4763-b03f-f3f118ab93b0"). InnerVolumeSpecName "kube-api-access-7jx4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:08:06.500011 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.499935    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:08:06.500011 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.500005    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:08:06.500011 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.500018    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jx4j\" (UniqueName: \"kubernetes.io/projected/f661a46c-ad1e-4763-b03f-f3f118ab93b0-kube-api-access-7jx4j\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:08:06.500261 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.500029    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f661a46c-ad1e-4763-b03f-f3f118ab93b0-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:08:06.500261 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.500039    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f661a46c-ad1e-4763-b03f-f3f118ab93b0-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:08:06.700661 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.700565    2584 generic.go:356] "Generic (PLEG): container finished" podID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerID="4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660" exitCode=0
Sep 19 09:08:06.700661 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.700640    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"
Sep 19 09:08:06.700661 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.700648    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerDied","Data":"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"}
Sep 19 09:08:06.700880 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.700738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l" event={"ID":"f661a46c-ad1e-4763-b03f-f3f118ab93b0","Type":"ContainerDied","Data":"1a009a479c7ae6b04f3c1bbaa93519f47467d980ae860c1ab4cf26dbdbc18671"}
Sep 19 09:08:06.700880 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.700765    2584 scope.go:117] "RemoveContainer" containerID="bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"
Sep 19 09:08:06.710360 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.710342    2584 scope.go:117] "RemoveContainer" containerID="4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"
Sep 19 09:08:06.718142 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.718117    2584 scope.go:117] "RemoveContainer" containerID="c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"
Sep 19 09:08:06.722796 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.722773    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:08:06.726945 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.726920    2584 scope.go:117] "RemoveContainer" containerID="bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"
Sep 19 09:08:06.727279 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:08:06.727257    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0\": container with ID starting with bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0 not found: ID does not exist" containerID="bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"
Sep 19 09:08:06.727372 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727292    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0"} err="failed to get container status \"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0\": rpc error: code = NotFound desc = could not find container \"bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0\": container with ID starting with bb569fce679c8df65939d35b14e7b6160dbbbd8de2ad494343fbe35ef48702a0 not found: ID does not exist"
Sep 19 09:08:06.727372 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727318    2584 scope.go:117] "RemoveContainer" containerID="4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"
Sep 19 09:08:06.727526 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727509    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-rwb8l"]
Sep 19 09:08:06.727636 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:08:06.727615    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660\": container with ID starting with 4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660 not found: ID does not exist" containerID="4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"
Sep 19 09:08:06.727674 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727642    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660"} err="failed to get container status \"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660\": rpc error: code = NotFound desc = could not find container \"4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660\": container with ID starting with 4ec5a4052e34c53704c3bf3f5b3234893ad5f4b49014e2d29e1a266aa437a660 not found: ID does not exist"
Sep 19 09:08:06.727674 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727658    2584 scope.go:117] "RemoveContainer" containerID="c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"
Sep 19 09:08:06.727917 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:08:06.727901    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a\": container with ID starting with c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a not found: ID does not exist" containerID="c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"
Sep 19 09:08:06.727977 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:06.727921    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a"} err="failed to get container status \"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a\": rpc error: code = NotFound desc = could not find container \"c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a\": container with ID starting with c85d7e99caf1fa182969918e280623326423d8e415fd9e2e19557ee36d9fe76a not found: ID does not exist"
Sep 19 09:08:07.114161 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114126    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:08:07.114501 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114489    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container"
Sep 19 09:08:07.114551 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114503    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container"
Sep 19 09:08:07.114551 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114519    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="storage-initializer"
Sep 19 09:08:07.114551 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114524    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="storage-initializer"
Sep 19 09:08:07.114551 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114533    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kube-rbac-proxy"
Sep 19 09:08:07.114551 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114539    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kube-rbac-proxy"
Sep 19 09:08:07.114713 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114606    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kube-rbac-proxy"
Sep 19 09:08:07.114713 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.114615    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" containerName="kserve-container"
Sep 19 09:08:07.119127 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.119099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.121801 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.121777    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:08:07.121967 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.121938    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 19 09:08:07.122073 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.122054    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 19 09:08:07.122134 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.122072    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:08:07.122194 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.122165    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:08:07.122194 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.122170    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:08:07.129718 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.129696    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:08:07.206109 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.206070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.206331 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.206138    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.206331 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.206200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bhz\" (UniqueName: \"kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.206331 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.206258    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.307197 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.307160    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bhz\" (UniqueName: \"kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.307197 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.307198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.307750 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.307241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.307750 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.307274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.307750 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.307571    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.308116 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.308094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.309751 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.309729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.315275 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.315256    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bhz\" (UniqueName: \"kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz\") pod \"isvc-sklearn-s3-predictor-79698dfdf9-9g2vz\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.431277 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.431184    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:07.705238 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.705200    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:08:07.707593 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:08:07.707566    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938db65d_5832_4d96_8a91_302e206895ff.slice/crio-0c2e1ca31400a50ab2e06119162185d60227bfa70c9294ca4dd6e3fdcd1050fc WatchSource:0}: Error finding container 0c2e1ca31400a50ab2e06119162185d60227bfa70c9294ca4dd6e3fdcd1050fc: Status 404 returned error can't find the container with id 0c2e1ca31400a50ab2e06119162185d60227bfa70c9294ca4dd6e3fdcd1050fc
Sep 19 09:08:07.908132 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:07.908100    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f661a46c-ad1e-4763-b03f-f3f118ab93b0" path="/var/lib/kubelet/pods/f661a46c-ad1e-4763-b03f-f3f118ab93b0/volumes"
Sep 19 09:08:08.717332 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:08.717294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerStarted","Data":"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"}
Sep 19 09:08:08.717745 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:08.717359    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerStarted","Data":"0c2e1ca31400a50ab2e06119162185d60227bfa70c9294ca4dd6e3fdcd1050fc"}
Sep 19 09:08:09.723177 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:09.723142    2584 generic.go:356] "Generic (PLEG): container finished" podID="938db65d-5832-4d96-8a91-302e206895ff" containerID="ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315" exitCode=0
Sep 19 09:08:09.723583 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:09.723213    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerDied","Data":"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"}
Sep 19 09:08:10.730698 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:10.730659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerStarted","Data":"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"}
Sep 19 09:08:10.731233 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:10.730726    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerStarted","Data":"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"}
Sep 19 09:08:10.731233 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:10.730805    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:10.753751 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:10.753696    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podStartSLOduration=2.7139301700000003 podStartE2EDuration="3.753678951s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.039748781s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:08:10.751338036 +0000 UTC m=+2858.835838864" watchObservedRunningTime="2026-09-19 09:08:10.753678951 +0000 UTC m=+2858.838179753"
Sep 19 09:08:11.735752 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:11.735709    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:11.737070 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:11.737046    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:12.741105 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:12.741065    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:17.746139 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:17.746067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:08:17.746529 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:17.746500    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:27.746861 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:27.746811    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:37.747142 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:37.747091    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:47.746530 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:47.746485    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:08:57.746562 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:08:57.746526    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:09:07.747105 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:07.747062    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:09:17.746998 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:17.746938    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:09:27.747136 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:27.747105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:09:37.640482 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:37.640442    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:09:37.640995 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:37.640739    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" containerID="cri-o://48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce" gracePeriod=30
Sep 19 09:09:37.640995 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:37.640776    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy" containerID="cri-o://0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017" gracePeriod=30
Sep 19 09:09:37.742034 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:37.741956    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Sep 19 09:09:37.747342 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:37.747297    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 19 09:09:38.171763 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:38.171729    2584 generic.go:356] "Generic (PLEG): container finished" podID="938db65d-5832-4d96-8a91-302e206895ff" containerID="0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017" exitCode=2
Sep 19 09:09:38.171969 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:38.171805    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerDied","Data":"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"}
Sep 19 09:09:42.741863 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:42.741812    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Sep 19 09:09:43.093087 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.093057    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:09:43.198768 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.198732    2584 generic.go:356] "Generic (PLEG): container finished" podID="938db65d-5832-4d96-8a91-302e206895ff" containerID="48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce" exitCode=0
Sep 19 09:09:43.198978 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.198794    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"
Sep 19 09:09:43.198978 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.198799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerDied","Data":"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"}
Sep 19 09:09:43.198978 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.198862    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz" event={"ID":"938db65d-5832-4d96-8a91-302e206895ff","Type":"ContainerDied","Data":"0c2e1ca31400a50ab2e06119162185d60227bfa70c9294ca4dd6e3fdcd1050fc"}
Sep 19 09:09:43.198978 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.198897    2584 scope.go:117] "RemoveContainer" containerID="0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"
Sep 19 09:09:43.207902 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.207883    2584 scope.go:117] "RemoveContainer" containerID="48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"
Sep 19 09:09:43.215829 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.215811    2584 scope.go:117] "RemoveContainer" containerID="ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"
Sep 19 09:09:43.223724 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.223706    2584 scope.go:117] "RemoveContainer" containerID="0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"
Sep 19 09:09:43.224078 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:09:43.224058    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017\": container with ID starting with 0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017 not found: ID does not exist" containerID="0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"
Sep 19 09:09:43.224166 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.224092    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017"} err="failed to get container status \"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017\": rpc error: code = NotFound desc = could not find container \"0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017\": container with ID starting with 0d212d131c050389cedb4240856de6d44505c122995668212352dd407fd1a017 not found: ID does not exist"
Sep 19 09:09:43.224166 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.224117    2584 scope.go:117] "RemoveContainer" containerID="48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"
Sep 19 09:09:43.224396 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:09:43.224378    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce\": container with ID starting with 48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce not found: ID does not exist" containerID="48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"
Sep 19 09:09:43.224440 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.224404    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce"} err="failed to get container status \"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce\": rpc error: code = NotFound desc = could not find container \"48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce\": container with ID starting with 48ef5c3e37765adb7154419e89605350b368ed2f7c8a120f2105dce488ff67ce not found: ID does not exist"
Sep 19 09:09:43.224440 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.224419    2584 scope.go:117] "RemoveContainer" containerID="ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"
Sep 19 09:09:43.224653 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:09:43.224634    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315\": container with ID starting with ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315 not found: ID does not exist" containerID="ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"
Sep 19 09:09:43.224710 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.224663    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315"} err="failed to get container status \"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315\": rpc error: code = NotFound desc = could not find container \"ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315\": container with ID starting with ebf141fc0bef52acde5625b09fac598dc2f5f71e36ba9a0ebad365a9b7bd6315 not found: ID does not exist"
Sep 19 09:09:43.270115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270029    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls\") pod \"938db65d-5832-4d96-8a91-302e206895ff\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") "
Sep 19 09:09:43.270115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270097    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config\") pod \"938db65d-5832-4d96-8a91-302e206895ff\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") "
Sep 19 09:09:43.270306 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270137    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location\") pod \"938db65d-5832-4d96-8a91-302e206895ff\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") "
Sep 19 09:09:43.270306 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270163    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2bhz\" (UniqueName: \"kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz\") pod \"938db65d-5832-4d96-8a91-302e206895ff\" (UID: \"938db65d-5832-4d96-8a91-302e206895ff\") "
Sep 19 09:09:43.270489 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270460    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "938db65d-5832-4d96-8a91-302e206895ff" (UID: "938db65d-5832-4d96-8a91-302e206895ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:43.270531 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.270484    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "938db65d-5832-4d96-8a91-302e206895ff" (UID: "938db65d-5832-4d96-8a91-302e206895ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:09:43.272269 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.272249    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "938db65d-5832-4d96-8a91-302e206895ff" (UID: "938db65d-5832-4d96-8a91-302e206895ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:09:43.272405 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.272388    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz" (OuterVolumeSpecName: "kube-api-access-j2bhz") pod "938db65d-5832-4d96-8a91-302e206895ff" (UID: "938db65d-5832-4d96-8a91-302e206895ff"). InnerVolumeSpecName "kube-api-access-j2bhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:43.370842 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.370801    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938db65d-5832-4d96-8a91-302e206895ff-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:09:43.370842 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.370835    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938db65d-5832-4d96-8a91-302e206895ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:09:43.370842 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.370849    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938db65d-5832-4d96-8a91-302e206895ff-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:09:43.371185 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.370861    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bhz\" (UniqueName: \"kubernetes.io/projected/938db65d-5832-4d96-8a91-302e206895ff-kube-api-access-j2bhz\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:09:43.532242 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.532144    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:09:43.539663 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.539631    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-79698dfdf9-9g2vz"]
Sep 19 09:09:43.907482 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:09:43.907401    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938db65d-5832-4d96-8a91-302e206895ff" path="/var/lib/kubelet/pods/938db65d-5832-4d96-8a91-302e206895ff/volumes"
Sep 19 09:10:34.166846 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:10:34.166816    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:10:34.172195 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:10:34.172169    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:10:34.174731 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:10:34.174706    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:10:34.185777 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:10:34.185757    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:11:18.956929 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.956850    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957253    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="storage-initializer"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957267    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="storage-initializer"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957288    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957294    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957308    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957314    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957378    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kube-rbac-proxy"
Sep 19 09:11:18.957391 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.957391    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="938db65d-5832-4d96-8a91-302e206895ff" containerName="kserve-container"
Sep 19 09:11:18.960709 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.960691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:18.963130 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963103    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:11:18.963291 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963271    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 19 09:11:18.963379 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963351    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 19 09:11:18.963685 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963669    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:11:18.963770 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963676    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:11:18.963955 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.963921    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:11:18.973401 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:18.973378    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:19.115938 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.115900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.116150 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.115956    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt4x\" (UniqueName: \"kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.116150 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.116124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.116258 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.116182    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.217358 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.217318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.217560 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.217386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.217560 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.217424    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.217560 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.217462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt4x\" (UniqueName: \"kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.217902 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.217871    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.218028 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.218009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.220059 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.220038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.226082 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.226065    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt4x\" (UniqueName: \"kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.272101 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.272065    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:19.544487 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.544459    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:19.547056 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:11:19.547031    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255ba306_5b12_4de8_8089_a7f7cda62eec.slice/crio-49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf WatchSource:0}: Error finding container 49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf: Status 404 returned error can't find the container with id 49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf
Sep 19 09:11:19.672804 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.672772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerStarted","Data":"f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"}
Sep 19 09:11:19.672959 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:19.672830    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerStarted","Data":"49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf"}
Sep 19 09:11:23.694101 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:23.694021    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/0.log"
Sep 19 09:11:23.694101 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:23.694057    2584 generic.go:356] "Generic (PLEG): container finished" podID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerID="f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9" exitCode=1
Sep 19 09:11:23.694101 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:23.694083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerDied","Data":"f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"}
Sep 19 09:11:24.701635 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:24.701605    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/0.log"
Sep 19 09:11:24.702048 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:24.701685    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerStarted","Data":"658f85ebc45a14051bf40868de7c8fdf773dcc95541c3ef619718053bbbd1598"}
Sep 19 09:11:26.712199 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.712119    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/1.log"
Sep 19 09:11:26.712572 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.712520    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/0.log"
Sep 19 09:11:26.712572 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.712554    2584 generic.go:356] "Generic (PLEG): container finished" podID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerID="658f85ebc45a14051bf40868de7c8fdf773dcc95541c3ef619718053bbbd1598" exitCode=1
Sep 19 09:11:26.712651 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.712628    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerDied","Data":"658f85ebc45a14051bf40868de7c8fdf773dcc95541c3ef619718053bbbd1598"}
Sep 19 09:11:26.712708 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.712695    2584 scope.go:117] "RemoveContainer" containerID="f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"
Sep 19 09:11:26.713106 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.713086    2584 scope.go:117] "RemoveContainer" containerID="f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"
Sep 19 09:11:26.723622 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:11:26.723597    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_kserve-ci-e2e-test(255ba306-5b12-4de8-8089-a7f7cda62eec)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec"
Sep 19 09:11:26.723696 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:11:26.723658    2584 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_kserve-ci-e2e-test_255ba306-5b12-4de8-8089-a7f7cda62eec_0 in pod sandbox 49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf from index: no such id: 'f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9'" containerID="f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"
Sep 19 09:11:26.723751 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:26.723697    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_kserve-ci-e2e-test_255ba306-5b12-4de8-8089-a7f7cda62eec_0 in pod sandbox 49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf from index: no such id: 'f75446e0afc32a48719dafddf69421c00158bce3128123d6552b1df5edfd3be9'"
Sep 19 09:11:27.719430 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:27.719404    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/1.log"
Sep 19 09:11:28.939872 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:28.939840    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:29.073349 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.073330    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/1.log"
Sep 19 09:11:29.077051 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.077034    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:29.104468 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104438    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config\") pod \"255ba306-5b12-4de8-8089-a7f7cda62eec\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") "
Sep 19 09:11:29.104589 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104481    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls\") pod \"255ba306-5b12-4de8-8089-a7f7cda62eec\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") "
Sep 19 09:11:29.104644 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104608    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flt4x\" (UniqueName: \"kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x\") pod \"255ba306-5b12-4de8-8089-a7f7cda62eec\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") "
Sep 19 09:11:29.104730 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location\") pod \"255ba306-5b12-4de8-8089-a7f7cda62eec\" (UID: \"255ba306-5b12-4de8-8089-a7f7cda62eec\") "
Sep 19 09:11:29.104840 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "255ba306-5b12-4de8-8089-a7f7cda62eec" (UID: "255ba306-5b12-4de8-8089-a7f7cda62eec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:29.104967 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.104945    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "255ba306-5b12-4de8-8089-a7f7cda62eec" (UID: "255ba306-5b12-4de8-8089-a7f7cda62eec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:29.105063 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.105044    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/255ba306-5b12-4de8-8089-a7f7cda62eec-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:11:29.105111 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.105069    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:11:29.106655 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.106629    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x" (OuterVolumeSpecName: "kube-api-access-flt4x") pod "255ba306-5b12-4de8-8089-a7f7cda62eec" (UID: "255ba306-5b12-4de8-8089-a7f7cda62eec"). InnerVolumeSpecName "kube-api-access-flt4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:11:29.106655 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.106647    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "255ba306-5b12-4de8-8089-a7f7cda62eec" (UID: "255ba306-5b12-4de8-8089-a7f7cda62eec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:11:29.206504 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.206430    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flt4x\" (UniqueName: \"kubernetes.io/projected/255ba306-5b12-4de8-8089-a7f7cda62eec-kube-api-access-flt4x\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:11:29.206504 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.206458    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/255ba306-5b12-4de8-8089-a7f7cda62eec-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:11:29.730026 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.730002    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz_255ba306-5b12-4de8-8089-a7f7cda62eec/storage-initializer/1.log"
Sep 19 09:11:29.730203 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.730098    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"
Sep 19 09:11:29.730203 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.730133    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz" event={"ID":"255ba306-5b12-4de8-8089-a7f7cda62eec","Type":"ContainerDied","Data":"49bbc28b260835cc30cd6148173f6a2a38e6cc20bba8b0a710aa377d7d6c63bf"}
Sep 19 09:11:29.730203 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.730196    2584 scope.go:117] "RemoveContainer" containerID="658f85ebc45a14051bf40868de7c8fdf773dcc95541c3ef619718053bbbd1598"
Sep 19 09:11:29.769372 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.769347    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:29.775842 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.775820    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-77c7fccb99-mp7xz"]
Sep 19 09:11:29.907110 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:29.907073    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" path="/var/lib/kubelet/pods/255ba306-5b12-4de8-8089-a7f7cda62eec/volumes"
Sep 19 09:11:30.093482 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.093404    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:11:30.093952 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.093937    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.094028 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.093954    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.094121 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.094109    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.094155 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.094126    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.098060 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.094823    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.098060 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.094855    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="255ba306-5b12-4de8-8089-a7f7cda62eec" containerName="storage-initializer"
Sep 19 09:11:30.102335 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.102312    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.105639 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.105609    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:11:30.105807 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.105784    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 19 09:11:30.106113 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.105860    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 19 09:11:30.106214 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.106111    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:11:30.106214 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.106180    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:11:30.106328 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.106219    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:11:30.106570 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.106550    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:11:30.113922 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.113899    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:11:30.115726 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.115704    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skj5k\" (UniqueName: \"kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.115845 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.115733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.115845 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.115769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.115845 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.115821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.116036 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.115931    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.216767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.216728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skj5k\" (UniqueName: \"kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.216767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.216771    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.216823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.216849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.216912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217115 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:11:30.216968    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 19 09:11:30.217115 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:11:30.217066    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls podName:6013c052-ca1b-4a11-a345-8634e422efa5 nodeName:}" failed. No retries permitted until 2026-09-19 09:11:30.71704427 +0000 UTC m=+3058.801545053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 19 09:11:30.217407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.217285    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217623 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.217600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.217692 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.217624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.226031 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.226009    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skj5k\" (UniqueName: \"kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.722867 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.722824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:30.725313 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:30.725280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:31.014606 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:31.014517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:31.312218 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:31.312197    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:11:31.315155 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:11:31.315124    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6013c052_ca1b_4a11_a345_8634e422efa5.slice/crio-f8b41b68a007a89f644719d03f82984f3e9675207e730bbab182b369d8718fa5 WatchSource:0}: Error finding container f8b41b68a007a89f644719d03f82984f3e9675207e730bbab182b369d8718fa5: Status 404 returned error can't find the container with id f8b41b68a007a89f644719d03f82984f3e9675207e730bbab182b369d8718fa5
Sep 19 09:11:31.745818 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:31.745781    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerStarted","Data":"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"}
Sep 19 09:11:31.745972 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:31.745853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerStarted","Data":"f8b41b68a007a89f644719d03f82984f3e9675207e730bbab182b369d8718fa5"}
Sep 19 09:11:32.753024 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:32.752906    2584 generic.go:356] "Generic (PLEG): container finished" podID="6013c052-ca1b-4a11-a345-8634e422efa5" containerID="b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487" exitCode=0
Sep 19 09:11:32.753024 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:32.753008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerDied","Data":"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"}
Sep 19 09:11:33.760919 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:33.760879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerStarted","Data":"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"}
Sep 19 09:11:33.761439 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:33.760959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerStarted","Data":"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"}
Sep 19 09:11:33.761439 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:33.761032    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:33.781732 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:33.781669    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podStartSLOduration=2.7057389020000002 podStartE2EDuration="3.781648207s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.075909305s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:11:33.779279765 +0000 UTC m=+3061.863780581" watchObservedRunningTime="2026-09-19 09:11:33.781648207 +0000 UTC m=+3061.866149010"
Sep 19 09:11:34.765826 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:34.765786    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:34.767490 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:34.767456    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:11:35.770487 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:35.770441    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:11:40.776348 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:40.776319    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:11:40.776844 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:40.776817    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:11:50.776821 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:11:50.776774    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:00.777384 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:00.777342    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:10.777658 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:10.777611    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:20.777691 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:20.777653    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:30.776892 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:30.776844    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:40.777323 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:40.777272    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:12:50.777736 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:12:50.777646    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:13:00.778184 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:00.778152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:13:10.086759 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.086722    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:13:10.087324 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.086974    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" containerID="cri-o://253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a" gracePeriod=30
Sep 19 09:13:10.087324 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.087052    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kube-rbac-proxy" containerID="cri-o://3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81" gracePeriod=30
Sep 19 09:13:10.234281 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.234249    2584 generic.go:356] "Generic (PLEG): container finished" podID="6013c052-ca1b-4a11-a345-8634e422efa5" containerID="3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81" exitCode=2
Sep 19 09:13:10.234454 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.234318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerDied","Data":"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"}
Sep 19 09:13:10.771637 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.771601    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
Sep 19 09:13:10.777019 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:10.776971    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
Sep 19 09:13:11.226720 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.226683    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:11.230406 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.230389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.232757 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.232735    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 19 09:13:11.232915 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.232896    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 19 09:13:11.242960 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.242933    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:11.349911 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.349877    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.350094 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.349921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.350094 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.350058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.350232 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.350102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.450604 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.450559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.450604 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.450605    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.450869 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.450637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.450869 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.450668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.451239 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.451214    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.451512 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.451495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.453140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.453116    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.458966 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.458945    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.544081 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.543962    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:11.818109 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:11.818082    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:11.820375 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:13:11.820345    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6ccdb1_4bfb_4db5_9aca_31175c521379.slice/crio-bb58d63cc2c4a09d75249afd9de563c7b9f611799e0f9a116f7e044e2298ffbd WatchSource:0}: Error finding container bb58d63cc2c4a09d75249afd9de563c7b9f611799e0f9a116f7e044e2298ffbd: Status 404 returned error can't find the container with id bb58d63cc2c4a09d75249afd9de563c7b9f611799e0f9a116f7e044e2298ffbd
Sep 19 09:13:12.249463 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:12.249429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerStarted","Data":"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"}
Sep 19 09:13:12.249862 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:12.249487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerStarted","Data":"bb58d63cc2c4a09d75249afd9de563c7b9f611799e0f9a116f7e044e2298ffbd"}
Sep 19 09:13:15.240495 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.240471    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:13:15.265998 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.265958    2584 generic.go:356] "Generic (PLEG): container finished" podID="6013c052-ca1b-4a11-a345-8634e422efa5" containerID="253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a" exitCode=0
Sep 19 09:13:15.266140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.266025    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerDied","Data":"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"}
Sep 19 09:13:15.266140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.266091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj" event={"ID":"6013c052-ca1b-4a11-a345-8634e422efa5","Type":"ContainerDied","Data":"f8b41b68a007a89f644719d03f82984f3e9675207e730bbab182b369d8718fa5"}
Sep 19 09:13:15.266140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.266032    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"
Sep 19 09:13:15.266140 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.266126    2584 scope.go:117] "RemoveContainer" containerID="3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"
Sep 19 09:13:15.274458 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.274436    2584 scope.go:117] "RemoveContainer" containerID="253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"
Sep 19 09:13:15.283181 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.283103    2584 scope.go:117] "RemoveContainer" containerID="b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"
Sep 19 09:13:15.284932 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.284902    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location\") pod \"6013c052-ca1b-4a11-a345-8634e422efa5\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") "
Sep 19 09:13:15.285060 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.284935    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") pod \"6013c052-ca1b-4a11-a345-8634e422efa5\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") "
Sep 19 09:13:15.285060 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285017    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skj5k\" (UniqueName: \"kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k\") pod \"6013c052-ca1b-4a11-a345-8634e422efa5\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") "
Sep 19 09:13:15.285174 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285075    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config\") pod \"6013c052-ca1b-4a11-a345-8634e422efa5\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") "
Sep 19 09:13:15.285174 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285127    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert\") pod \"6013c052-ca1b-4a11-a345-8634e422efa5\" (UID: \"6013c052-ca1b-4a11-a345-8634e422efa5\") "
Sep 19 09:13:15.285298 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285178    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6013c052-ca1b-4a11-a345-8634e422efa5" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:13:15.285471 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285449    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6013c052-ca1b-4a11-a345-8634e422efa5" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:13:15.285560 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285470    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6013c052-ca1b-4a11-a345-8634e422efa5-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:15.285621 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.285602    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6013c052-ca1b-4a11-a345-8634e422efa5" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:13:15.286974 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.286951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6013c052-ca1b-4a11-a345-8634e422efa5" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:13:15.287308 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.287275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k" (OuterVolumeSpecName: "kube-api-access-skj5k") pod "6013c052-ca1b-4a11-a345-8634e422efa5" (UID: "6013c052-ca1b-4a11-a345-8634e422efa5"). InnerVolumeSpecName "kube-api-access-skj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:13:15.307307 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.307279    2584 scope.go:117] "RemoveContainer" containerID="3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"
Sep 19 09:13:15.307656 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:15.307630    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81\": container with ID starting with 3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81 not found: ID does not exist" containerID="3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"
Sep 19 09:13:15.307746 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.307662    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81"} err="failed to get container status \"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81\": rpc error: code = NotFound desc = could not find container \"3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81\": container with ID starting with 3b4a84dc64706b976539e4b0c80b765130b26a19f067a708e5d8008b76f78c81 not found: ID does not exist"
Sep 19 09:13:15.307746 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.307678    2584 scope.go:117] "RemoveContainer" containerID="253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"
Sep 19 09:13:15.307972 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:15.307943    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a\": container with ID starting with 253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a not found: ID does not exist" containerID="253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"
Sep 19 09:13:15.308092 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.307975    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a"} err="failed to get container status \"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a\": rpc error: code = NotFound desc = could not find container \"253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a\": container with ID starting with 253367d283ba71baaac09467094c6364958504e3bbeb4f491cd9a96d2ec43d9a not found: ID does not exist"
Sep 19 09:13:15.308092 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.308008    2584 scope.go:117] "RemoveContainer" containerID="b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"
Sep 19 09:13:15.308328 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:15.308306    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487\": container with ID starting with b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487 not found: ID does not exist" containerID="b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"
Sep 19 09:13:15.308414 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.308333    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487"} err="failed to get container status \"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487\": rpc error: code = NotFound desc = could not find container \"b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487\": container with ID starting with b5dc55ca738383258acf3c0af393e14fe30db1f34d28ea96b7b32d84b903c487 not found: ID does not exist"
Sep 19 09:13:15.386581 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.386496    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:15.386581 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.386527    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6013c052-ca1b-4a11-a345-8634e422efa5-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:15.386581 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.386538    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skj5k\" (UniqueName: \"kubernetes.io/projected/6013c052-ca1b-4a11-a345-8634e422efa5-kube-api-access-skj5k\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:15.386581 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.386547    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6013c052-ca1b-4a11-a345-8634e422efa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:15.588738 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.588708    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:13:15.592381 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.592356    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-658d5f6fb7-j7glj"]
Sep 19 09:13:15.906491 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:15.906457    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" path="/var/lib/kubelet/pods/6013c052-ca1b-4a11-a345-8634e422efa5/volumes"
Sep 19 09:13:16.274103 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:16.274074    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/0.log"
Sep 19 09:13:16.274489 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:16.274111    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerID="29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c" exitCode=1
Sep 19 09:13:16.274489 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:16.274149    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerDied","Data":"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"}
Sep 19 09:13:17.281008 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:17.280960    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/0.log"
Sep 19 09:13:17.281394 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:17.281087    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerStarted","Data":"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"}
Sep 19 09:13:21.183837 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.183799    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:21.184376 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.184095    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer" containerID="cri-o://746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf" gracePeriod=30
Sep 19 09:13:21.840872 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.840847    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/1.log"
Sep 19 09:13:21.841302 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.841285    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/0.log"
Sep 19 09:13:21.843639 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.843623    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:21.946613 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.946523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location\") pod \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") "
Sep 19 09:13:21.946613 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.946572    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls\") pod \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") "
Sep 19 09:13:21.946860 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.946642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc\") pod \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") "
Sep 19 09:13:21.946860 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.946663    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config\") pod \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\" (UID: \"ff6ccdb1-4bfb-4db5-9aca-31175c521379\") "
Sep 19 09:13:21.947005 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.946868    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff6ccdb1-4bfb-4db5-9aca-31175c521379" (UID: "ff6ccdb1-4bfb-4db5-9aca-31175c521379"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:13:21.947093 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.947067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff6ccdb1-4bfb-4db5-9aca-31175c521379" (UID: "ff6ccdb1-4bfb-4db5-9aca-31175c521379"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:13:21.948890 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.948867    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff6ccdb1-4bfb-4db5-9aca-31175c521379" (UID: "ff6ccdb1-4bfb-4db5-9aca-31175c521379"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:13:21.949004 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:21.948917    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc" (OuterVolumeSpecName: "kube-api-access-j4pkc") pod "ff6ccdb1-4bfb-4db5-9aca-31175c521379" (UID: "ff6ccdb1-4bfb-4db5-9aca-31175c521379"). InnerVolumeSpecName "kube-api-access-j4pkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:13:22.048255 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.048214    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:22.048255 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.048245    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6ccdb1-4bfb-4db5-9aca-31175c521379-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:22.048255 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.048256    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pkc\" (UniqueName: \"kubernetes.io/projected/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-api-access-j4pkc\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:22.048509 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.048268    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6ccdb1-4bfb-4db5-9aca-31175c521379-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:13:22.308053 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308028    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/1.log"
Sep 19 09:13:22.308482 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df_ff6ccdb1-4bfb-4db5-9aca-31175c521379/storage-initializer/0.log"
Sep 19 09:13:22.308528 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308502    2584 generic.go:356] "Generic (PLEG): container finished" podID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerID="746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf" exitCode=1
Sep 19 09:13:22.308574 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308559    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"
Sep 19 09:13:22.308574 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308560    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerDied","Data":"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"}
Sep 19 09:13:22.308646 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df" event={"ID":"ff6ccdb1-4bfb-4db5-9aca-31175c521379","Type":"ContainerDied","Data":"bb58d63cc2c4a09d75249afd9de563c7b9f611799e0f9a116f7e044e2298ffbd"}
Sep 19 09:13:22.308680 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.308662    2584 scope.go:117] "RemoveContainer" containerID="746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"
Sep 19 09:13:22.318914 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.318892    2584 scope.go:117] "RemoveContainer" containerID="29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"
Sep 19 09:13:22.323452 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323430    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:13:22.323846 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323829    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.323846 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323846    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323856    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323862    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323878    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="storage-initializer"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323886    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="storage-initializer"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323899    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kube-rbac-proxy"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323907    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kube-rbac-proxy"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323920    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container"
Sep 19 09:13:22.324021 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.323925    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container"
Sep 19 09:13:22.324418 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.324081    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.324418 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.324103    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kube-rbac-proxy"
Sep 19 09:13:22.324418 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.324110    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6013c052-ca1b-4a11-a345-8634e422efa5" containerName="kserve-container"
Sep 19 09:13:22.324418 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.324247    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" containerName="storage-initializer"
Sep 19 09:13:22.328414 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.328397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.328585 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.328566    2584 scope.go:117] "RemoveContainer" containerID="746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"
Sep 19 09:13:22.328904 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:22.328880    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf\": container with ID starting with 746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf not found: ID does not exist" containerID="746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"
Sep 19 09:13:22.329028 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.328913    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf"} err="failed to get container status \"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf\": rpc error: code = NotFound desc = could not find container \"746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf\": container with ID starting with 746269ab3e1526cf76867d7a92ac5222f3911e90bf465531e54c7a2b51d95fdf not found: ID does not exist"
Sep 19 09:13:22.329028 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.328948    2584 scope.go:117] "RemoveContainer" containerID="29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"
Sep 19 09:13:22.329275 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:22.329257    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c\": container with ID starting with 29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c not found: ID does not exist" containerID="29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"
Sep 19 09:13:22.329343 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.329281    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c"} err="failed to get container status \"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c\": rpc error: code = NotFound desc = could not find container \"29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c\": container with ID starting with 29bda75a87cda720a973ecd3de30043631bc39a2dab65a23c1d99fef3872e67c not found: ID does not exist"
Sep 19 09:13:22.334190 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334159    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:13:22.334437 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334421    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 19 09:13:22.334509 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334421    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 19 09:13:22.334509 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334489    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 19 09:13:22.334655 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334639    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:13:22.334722 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334673    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:13:22.334722 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.334715    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:13:22.346348 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.346318    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:13:22.360587 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.360560    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:22.363285 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.363207    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5957446b95-869df"]
Sep 19 09:13:22.452220 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.452179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.452407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.452231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk79z\" (UniqueName: \"kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.452407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.452318    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.452407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.452362    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.452407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.452396    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.552951 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.552908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.552958    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553025    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:22.553096    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk79z\" (UniqueName: \"kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553151 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:22.553155    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls podName:8626bec5-803f-4ad3-b711-60e5af912f37 nodeName:}" failed. No retries permitted until 2026-09-19 09:13:23.053139684 +0000 UTC m=+3171.137640469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 19 09:13:22.553478 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553457    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553778 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553757    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.553875 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.553846    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:22.567042 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:22.563912    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk79z\" (UniqueName: \"kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:23.057016 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:23.056959    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:23.057216 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:23.057114    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 19 09:13:23.057216 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:13:23.057188    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls podName:8626bec5-803f-4ad3-b711-60e5af912f37 nodeName:}" failed. No retries permitted until 2026-09-19 09:13:24.057172826 +0000 UTC m=+3172.141673612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 19 09:13:23.908090 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:23.908056    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6ccdb1-4bfb-4db5-9aca-31175c521379" path="/var/lib/kubelet/pods/ff6ccdb1-4bfb-4db5-9aca-31175c521379/volumes"
Sep 19 09:13:24.065320 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:24.065284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:24.067852 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:24.067820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:24.139501 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:24.139462    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:24.423621 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:24.423595    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:13:24.424928 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:13:24.424901    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8626bec5_803f_4ad3_b711_60e5af912f37.slice/crio-31b889adef2f37c88e941ee3ea6e423b17af347da4b40e283c10d075f44a673c WatchSource:0}: Error finding container 31b889adef2f37c88e941ee3ea6e423b17af347da4b40e283c10d075f44a673c: Status 404 returned error can't find the container with id 31b889adef2f37c88e941ee3ea6e423b17af347da4b40e283c10d075f44a673c
Sep 19 09:13:25.326915 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:25.326871    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerStarted","Data":"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"}
Sep 19 09:13:25.327479 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:25.326939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerStarted","Data":"31b889adef2f37c88e941ee3ea6e423b17af347da4b40e283c10d075f44a673c"}
Sep 19 09:13:26.334068 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:26.334026    2584 generic.go:356] "Generic (PLEG): container finished" podID="8626bec5-803f-4ad3-b711-60e5af912f37" containerID="a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f" exitCode=0
Sep 19 09:13:26.334530 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:26.334109    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerDied","Data":"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"}
Sep 19 09:13:27.341096 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:27.341057    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerStarted","Data":"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"}
Sep 19 09:13:27.341483 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:27.341130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerStarted","Data":"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"}
Sep 19 09:13:27.341483 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:27.341205    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:27.368739 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:27.368687    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podStartSLOduration=4.309082726 podStartE2EDuration="5.368670773s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.059588047s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:13:27.367812221 +0000 UTC m=+3175.452313036" watchObservedRunningTime="2026-09-19 09:13:27.368670773 +0000 UTC m=+3175.453171635"
Sep 19 09:13:28.346545 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:28.346501    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:28.347925 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:28.347896    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:13:29.351730 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:29.351669    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:13:34.356713 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:34.356681    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:13:34.357318 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:34.357286    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:13:44.357355 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:44.357314    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:13:54.357701 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:13:54.357654    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:04.357202 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:04.357165    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:14.357294 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:14.357252    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:24.357826 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:24.357736    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:34.357975 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:34.357934    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:43.903760 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:43.903714    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:14:53.907210 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:14:53.907166    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:15:02.337185 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:02.337144    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:15:02.337884 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:02.337447    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" containerID="cri-o://f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1" gracePeriod=30
Sep 19 09:15:02.337884 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:02.337496    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kube-rbac-proxy" containerID="cri-o://a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920" gracePeriod=30
Sep 19 09:15:02.834784 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:02.834753    2584 generic.go:356] "Generic (PLEG): container finished" podID="8626bec5-803f-4ad3-b711-60e5af912f37" containerID="a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920" exitCode=2
Sep 19 09:15:02.834935 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:02.834827    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerDied","Data":"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"}
Sep 19 09:15:03.903774 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:03.903725    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 19 09:15:04.352317 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:04.352275    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Sep 19 09:15:07.575439 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.575414    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:15:07.708218 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708126    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk79z\" (UniqueName: \"kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z\") pod \"8626bec5-803f-4ad3-b711-60e5af912f37\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") "
Sep 19 09:15:07.708218 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708180    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config\") pod \"8626bec5-803f-4ad3-b711-60e5af912f37\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") "
Sep 19 09:15:07.708410 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708265    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert\") pod \"8626bec5-803f-4ad3-b711-60e5af912f37\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") "
Sep 19 09:15:07.708410 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708303    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location\") pod \"8626bec5-803f-4ad3-b711-60e5af912f37\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") "
Sep 19 09:15:07.708410 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708367    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") pod \"8626bec5-803f-4ad3-b711-60e5af912f37\" (UID: \"8626bec5-803f-4ad3-b711-60e5af912f37\") "
Sep 19 09:15:07.708721 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8626bec5-803f-4ad3-b711-60e5af912f37" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:07.708721 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708705    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8626bec5-803f-4ad3-b711-60e5af912f37" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:15:07.708881 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.708710    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8626bec5-803f-4ad3-b711-60e5af912f37" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:15:07.710454 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.710430    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z" (OuterVolumeSpecName: "kube-api-access-sk79z") pod "8626bec5-803f-4ad3-b711-60e5af912f37" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37"). InnerVolumeSpecName "kube-api-access-sk79z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:15:07.710559 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.710478    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8626bec5-803f-4ad3-b711-60e5af912f37" (UID: "8626bec5-803f-4ad3-b711-60e5af912f37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:15:07.809767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.809736    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:15:07.809767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.809763    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8626bec5-803f-4ad3-b711-60e5af912f37-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:15:07.809767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.809774    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8626bec5-803f-4ad3-b711-60e5af912f37-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:15:07.810012 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.809784    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8626bec5-803f-4ad3-b711-60e5af912f37-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:15:07.810012 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.809794    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk79z\" (UniqueName: \"kubernetes.io/projected/8626bec5-803f-4ad3-b711-60e5af912f37-kube-api-access-sk79z\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:15:07.860130 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.860100    2584 generic.go:356] "Generic (PLEG): container finished" podID="8626bec5-803f-4ad3-b711-60e5af912f37" containerID="f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1" exitCode=0
Sep 19 09:15:07.860277 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.860154    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"
Sep 19 09:15:07.860277 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.860180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerDied","Data":"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"}
Sep 19 09:15:07.860277 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.860241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc" event={"ID":"8626bec5-803f-4ad3-b711-60e5af912f37","Type":"ContainerDied","Data":"31b889adef2f37c88e941ee3ea6e423b17af347da4b40e283c10d075f44a673c"}
Sep 19 09:15:07.860277 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.860268    2584 scope.go:117] "RemoveContainer" containerID="a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"
Sep 19 09:15:07.869423 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.869405    2584 scope.go:117] "RemoveContainer" containerID="f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"
Sep 19 09:15:07.877342 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.877325    2584 scope.go:117] "RemoveContainer" containerID="a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"
Sep 19 09:15:07.882610 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.882587    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:15:07.886132 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.886120    2584 scope.go:117] "RemoveContainer" containerID="a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"
Sep 19 09:15:07.886468 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:15:07.886434    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920\": container with ID starting with a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920 not found: ID does not exist" containerID="a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"
Sep 19 09:15:07.886563 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.886479    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920"} err="failed to get container status \"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920\": rpc error: code = NotFound desc = could not find container \"a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920\": container with ID starting with a256cfddf58874d8f549c11d60dbedab12137d663a9f59fcd779fe13f7fba920 not found: ID does not exist"
Sep 19 09:15:07.886563 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.886503    2584 scope.go:117] "RemoveContainer" containerID="f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"
Sep 19 09:15:07.886822 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:15:07.886801    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1\": container with ID starting with f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1 not found: ID does not exist" containerID="f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"
Sep 19 09:15:07.886862 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.886829    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1"} err="failed to get container status \"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1\": rpc error: code = NotFound desc = could not find container \"f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1\": container with ID starting with f8441ad62e1197fb779700453d80f0d810aa38f1445ba05a3efb1785c04d1bc1 not found: ID does not exist"
Sep 19 09:15:07.886862 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.886848    2584 scope.go:117] "RemoveContainer" containerID="a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"
Sep 19 09:15:07.887155 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:15:07.887134    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f\": container with ID starting with a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f not found: ID does not exist" containerID="a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"
Sep 19 09:15:07.887223 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.887161    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f"} err="failed to get container status \"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f\": rpc error: code = NotFound desc = could not find container \"a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f\": container with ID starting with a814a60864b29b7229ae863a15c23977335eeda6d44a450f384dee7bb0c2f54f not found: ID does not exist"
Sep 19 09:15:07.887264 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.887221    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5585664b8-l5qdc"]
Sep 19 09:15:07.907769 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:07.907742    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" path="/var/lib/kubelet/pods/8626bec5-803f-4ad3-b711-60e5af912f37/volumes"
Sep 19 09:15:14.576233 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576192    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576574    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kube-rbac-proxy"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576586    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kube-rbac-proxy"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576598    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576604    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576623    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="storage-initializer"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576628    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="storage-initializer"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576689    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kserve-container"
Sep 19 09:15:14.576827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.576701    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8626bec5-803f-4ad3-b711-60e5af912f37" containerName="kube-rbac-proxy"
Sep 19 09:15:14.581748 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.581728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.584588 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.584558    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qhq4r\""
Sep 19 09:15:14.585436 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.585410    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 09:15:14.585994 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.585964    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 09:15:14.586883 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.586857    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 09:15:14.587167 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.587147    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:15:14.587280 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.587261    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-8c0193-kube-rbac-proxy-sar-config\""
Sep 19 09:15:14.587661 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.587641    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-8c0193-predictor-serving-cert\""
Sep 19 09:15:14.591849 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.591806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:15:14.673075 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.673022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.673075 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.673079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.673327 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.673174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.673327 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.673229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.673327 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.673261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mcj\" (UniqueName: \"kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774010 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.773940    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774208 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774208 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774065    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mcj\" (UniqueName: \"kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774208 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774101    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774208 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774438 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:15:14.774228    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-8c0193-predictor-serving-cert: secret "isvc-primary-8c0193-predictor-serving-cert" not found
Sep 19 09:15:14.774438 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:15:14.774312    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls podName:e463b00c-2a9a-418c-877f-260140996954 nodeName:}" failed. No retries permitted until 2026-09-19 09:15:15.274291435 +0000 UTC m=+3283.358792228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls") pod "isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" (UID: "e463b00c-2a9a-418c-877f-260140996954") : secret "isvc-primary-8c0193-predictor-serving-cert" not found
Sep 19 09:15:14.774559 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774844 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774822    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.774925 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.774826    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:14.786147 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:14.786122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mcj\" (UniqueName: \"kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:15.278202 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.278167    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:15.280719 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.280688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") pod \"isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") " pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:15.501703 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.501653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:15.775827 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.775790    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:15:15.778238 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:15:15.778206    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode463b00c_2a9a_418c_877f_260140996954.slice/crio-6c028ba795a028a3e6a3f411770353430255bd8d80f88941b7456f185530c22e WatchSource:0}: Error finding container 6c028ba795a028a3e6a3f411770353430255bd8d80f88941b7456f185530c22e: Status 404 returned error can't find the container with id 6c028ba795a028a3e6a3f411770353430255bd8d80f88941b7456f185530c22e
Sep 19 09:15:15.907125 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.907091    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerStarted","Data":"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"}
Sep 19 09:15:15.907304 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:15.907151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerStarted","Data":"6c028ba795a028a3e6a3f411770353430255bd8d80f88941b7456f185530c22e"}
Sep 19 09:15:19.924635 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:19.924600    2584 generic.go:356] "Generic (PLEG): container finished" podID="e463b00c-2a9a-418c-877f-260140996954" containerID="fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1" exitCode=0
Sep 19 09:15:19.925035 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:19.924673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerDied","Data":"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"}
Sep 19 09:15:20.931877 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:20.931835    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerStarted","Data":"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"}
Sep 19 09:15:20.932337 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:20.931903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerStarted","Data":"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"}
Sep 19 09:15:20.932337 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:20.932110    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:21.937295 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:21.937256    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:21.938857 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:21.938826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:15:22.942436 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:22.942388    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:15:27.948587 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:27.948554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:15:27.949209 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:27.949088    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:15:27.969060 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:27.969010    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podStartSLOduration=10.09433941 podStartE2EDuration="13.968993811s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874654401s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:15:20.956252279 +0000 UTC m=+3289.040753085" watchObservedRunningTime="2026-09-19 09:15:27.968993811 +0000 UTC m=+3296.053494605"
Sep 19 09:15:34.206964 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:34.206936    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:15:34.219542 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:34.219511    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:15:34.222527 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:34.222503    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:15:34.227683 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:34.227659    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:15:37.949789 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:37.949747    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:15:47.949601 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:47.949518    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:15:57.949454 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:15:57.949407    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:16:07.949036 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:16:07.948975    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:16:17.949201 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:16:17.949160    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:16:27.949767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:16:27.949724    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:16:37.949619 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:16:37.949564    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 19 09:16:45.907188 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:16:45.907157    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:17:27.823242 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:27.823163    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:17:27.823769 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:27.823465    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container" containerID="cri-o://15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc" gracePeriod=30
Sep 19 09:17:27.823769 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:27.823488    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy" containerID="cri-o://69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46" gracePeriod=30
Sep 19 09:17:27.943396 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:27.943362    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Sep 19 09:17:28.550883 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:28.550843    2584 generic.go:356] "Generic (PLEG): container finished" podID="e463b00c-2a9a-418c-877f-260140996954" containerID="69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46" exitCode=2
Sep 19 09:17:28.551095 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:28.550918    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerDied","Data":"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"}
Sep 19 09:17:32.970439 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:32.970414    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:17:33.007103 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007069    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") pod \"e463b00c-2a9a-418c-877f-260140996954\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") "
Sep 19 09:17:33.007313 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007137    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location\") pod \"e463b00c-2a9a-418c-877f-260140996954\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") "
Sep 19 09:17:33.007313 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007161    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mcj\" (UniqueName: \"kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj\") pod \"e463b00c-2a9a-418c-877f-260140996954\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") "
Sep 19 09:17:33.007313 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007182    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert\") pod \"e463b00c-2a9a-418c-877f-260140996954\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") "
Sep 19 09:17:33.007313 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config\") pod \"e463b00c-2a9a-418c-877f-260140996954\" (UID: \"e463b00c-2a9a-418c-877f-260140996954\") "
Sep 19 09:17:33.007577 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007444    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e463b00c-2a9a-418c-877f-260140996954" (UID: "e463b00c-2a9a-418c-877f-260140996954"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:33.007639 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007604    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e463b00c-2a9a-418c-877f-260140996954" (UID: "e463b00c-2a9a-418c-877f-260140996954"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:17:33.007700 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.007683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e463b00c-2a9a-418c-877f-260140996954" (UID: "e463b00c-2a9a-418c-877f-260140996954"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:17:33.009339 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.009308    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj" (OuterVolumeSpecName: "kube-api-access-t5mcj") pod "e463b00c-2a9a-418c-877f-260140996954" (UID: "e463b00c-2a9a-418c-877f-260140996954"). InnerVolumeSpecName "kube-api-access-t5mcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:17:33.009455 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.009387    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e463b00c-2a9a-418c-877f-260140996954" (UID: "e463b00c-2a9a-418c-877f-260140996954"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:17:33.108664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.108576    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:17:33.108664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.108607    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e463b00c-2a9a-418c-877f-260140996954-proxy-tls\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:17:33.108664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.108619    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e463b00c-2a9a-418c-877f-260140996954-kserve-provision-location\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:17:33.108664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.108628    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mcj\" (UniqueName: \"kubernetes.io/projected/e463b00c-2a9a-418c-877f-260140996954-kube-api-access-t5mcj\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:17:33.108664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.108637    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e463b00c-2a9a-418c-877f-260140996954-cabundle-cert\") on node \"ip-10-0-136-99.ec2.internal\" DevicePath \"\""
Sep 19 09:17:33.579139 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.579103    2584 generic.go:356] "Generic (PLEG): container finished" podID="e463b00c-2a9a-418c-877f-260140996954" containerID="15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc" exitCode=0
Sep 19 09:17:33.579398 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.579158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerDied","Data":"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"}
Sep 19 09:17:33.579398 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.579164    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"
Sep 19 09:17:33.579398 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.579207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" event={"ID":"e463b00c-2a9a-418c-877f-260140996954","Type":"ContainerDied","Data":"6c028ba795a028a3e6a3f411770353430255bd8d80f88941b7456f185530c22e"}
Sep 19 09:17:33.579398 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.579233    2584 scope.go:117] "RemoveContainer" containerID="69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"
Sep 19 09:17:33.588468 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.588446    2584 scope.go:117] "RemoveContainer" containerID="15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"
Sep 19 09:17:33.596275 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.596259    2584 scope.go:117] "RemoveContainer" containerID="fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"
Sep 19 09:17:33.602156 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.602134    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:17:33.605101 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.605078    2584 scope.go:117] "RemoveContainer" containerID="69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"
Sep 19 09:17:33.605380 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:17:33.605359    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46\": container with ID starting with 69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46 not found: ID does not exist" containerID="69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"
Sep 19 09:17:33.605462 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.605392    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46"} err="failed to get container status \"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46\": rpc error: code = NotFound desc = could not find container \"69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46\": container with ID starting with 69a7c1e956d4ace88a3da196d9eaeb8bac27281905a35cabcfe648b96a494d46 not found: ID does not exist"
Sep 19 09:17:33.605462 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.605419    2584 scope.go:117] "RemoveContainer" containerID="15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"
Sep 19 09:17:33.605683 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:17:33.605667    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc\": container with ID starting with 15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc not found: ID does not exist" containerID="15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"
Sep 19 09:17:33.605748 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.605690    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc"} err="failed to get container status \"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc\": rpc error: code = NotFound desc = could not find container \"15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc\": container with ID starting with 15a9c5aaa70ec70c5c7b4d14d9da88b5e4b2d13a0ece68906022a1e92f8a46fc not found: ID does not exist"
Sep 19 09:17:33.605748 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.605711    2584 scope.go:117] "RemoveContainer" containerID="fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"
Sep 19 09:17:33.605978 ip-10-0-136-99 kubenswrapper[2584]: E0919 09:17:33.605956    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1\": container with ID starting with fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1 not found: ID does not exist" containerID="fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"
Sep 19 09:17:33.606051 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.606000    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1"} err="failed to get container status \"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1\": rpc error: code = NotFound desc = could not find container \"fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1\": container with ID starting with fd113c7a1b9eb845603ec802935a632545f56eb7c442193f5559b5c9d6fe03a1 not found: ID does not exist"
Sep 19 09:17:33.606130 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.606117    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4"]
Sep 19 09:17:33.909670 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.909594    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e463b00c-2a9a-418c-877f-260140996954" path="/var/lib/kubelet/pods/e463b00c-2a9a-418c-877f-260140996954/volumes"
Sep 19 09:17:33.942957 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:17:33.942910    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-8c0193-predictor-8bbdd55d7-hzzm4" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": context deadline exceeded"
Sep 19 09:18:51.248577 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248541    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-582w2/must-gather-vzbfx"]
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248907    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy"
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248920    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy"
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248938    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container"
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248943    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container"
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248959    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="storage-initializer"
Sep 19 09:18:51.249018 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.248965    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="storage-initializer"
Sep 19 09:18:51.249229 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.249040    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kube-rbac-proxy"
Sep 19 09:18:51.249229 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.249048    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e463b00c-2a9a-418c-877f-260140996954" containerName="kserve-container"
Sep 19 09:18:51.252252 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.252236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.254488 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.254463    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-582w2\"/\"default-dockercfg-phx2h\""
Sep 19 09:18:51.254595 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.254523    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-582w2\"/\"openshift-service-ca.crt\""
Sep 19 09:18:51.254595 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.254552    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-582w2\"/\"kube-root-ca.crt\""
Sep 19 09:18:51.261504 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.261478    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/must-gather-vzbfx"]
Sep 19 09:18:51.321314 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.321283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54x67\" (UniqueName: \"kubernetes.io/projected/3bd38c6b-f802-4d55-90ab-40d519b0628e-kube-api-access-54x67\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.321314 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.321323    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3bd38c6b-f802-4d55-90ab-40d519b0628e-must-gather-output\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.422362 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.422314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54x67\" (UniqueName: \"kubernetes.io/projected/3bd38c6b-f802-4d55-90ab-40d519b0628e-kube-api-access-54x67\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.422560 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.422375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3bd38c6b-f802-4d55-90ab-40d519b0628e-must-gather-output\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.422767 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.422744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3bd38c6b-f802-4d55-90ab-40d519b0628e-must-gather-output\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.430625 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.430593    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54x67\" (UniqueName: \"kubernetes.io/projected/3bd38c6b-f802-4d55-90ab-40d519b0628e-kube-api-access-54x67\") pod \"must-gather-vzbfx\" (UID: \"3bd38c6b-f802-4d55-90ab-40d519b0628e\") " pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.573672 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.573634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/must-gather-vzbfx"
Sep 19 09:18:51.846606 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.846521    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/must-gather-vzbfx"]
Sep 19 09:18:51.851240 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:18:51.851207    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd38c6b_f802_4d55_90ab_40d519b0628e.slice/crio-6318a941bbc6a0fd40764a87a03e797b359895323743c0fc5d25af6f4757b1ff WatchSource:0}: Error finding container 6318a941bbc6a0fd40764a87a03e797b359895323743c0fc5d25af6f4757b1ff: Status 404 returned error can't find the container with id 6318a941bbc6a0fd40764a87a03e797b359895323743c0fc5d25af6f4757b1ff
Sep 19 09:18:51.853195 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.853177    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:18:51.950821 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:51.950783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/must-gather-vzbfx" event={"ID":"3bd38c6b-f802-4d55-90ab-40d519b0628e","Type":"ContainerStarted","Data":"6318a941bbc6a0fd40764a87a03e797b359895323743c0fc5d25af6f4757b1ff"}
Sep 19 09:18:52.968309 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:52.968264    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/must-gather-vzbfx" event={"ID":"3bd38c6b-f802-4d55-90ab-40d519b0628e","Type":"ContainerStarted","Data":"5ab3814851c303e3cda96e4daca14d1cf0833b852a0ecc7c1952cef21e68016b"}
Sep 19 09:18:52.968701 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:52.968322    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/must-gather-vzbfx" event={"ID":"3bd38c6b-f802-4d55-90ab-40d519b0628e","Type":"ContainerStarted","Data":"d3d43bcfa91310c2dd4651f620fdf3ff911807be6409aad90272f291f4cf04a5"}
Sep 19 09:18:52.986714 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:52.986656    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-582w2/must-gather-vzbfx" podStartSLOduration=1.2774237689999999 podStartE2EDuration="1.986640208s" totalImagesPullingTime="709.216439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:18:52.983351369 +0000 UTC m=+3501.067852175" watchObservedRunningTime="2026-09-19 09:18:52.986640208 +0000 UTC m=+3501.071141010"
Sep 19 09:18:54.187688 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:54.187660    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j6tqd_c4775e37-d3b3-414d-a623-177de1acbfc0/global-pull-secret-syncer/0.log"
Sep 19 09:18:54.385869 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:54.385835    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbq2l_6171bdcd-0b75-4c34-8d84-9c28cfd810a0/konnectivity-agent/0.log"
Sep 19 09:18:54.436848 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:54.436814    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-99.ec2.internal_cc3207445c8278099d149a5cf6a8c10f/haproxy/0.log"
Sep 19 09:18:58.356441 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.356413    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/alertmanager/0.log"
Sep 19 09:18:58.393923 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.393850    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/config-reloader/0.log"
Sep 19 09:18:58.438032 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.438004    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/kube-rbac-proxy-web/0.log"
Sep 19 09:18:58.473285 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.473257    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/kube-rbac-proxy/0.log"
Sep 19 09:18:58.514061 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.514032    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/kube-rbac-proxy-metric/0.log"
Sep 19 09:18:58.551113 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.551085    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/prom-label-proxy/0.log"
Sep 19 09:18:58.589294 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.588751    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59e95e83-4c02-488c-8f19-917b84184cca/init-config-reloader/0.log"
Sep 19 09:18:58.645269 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.645179    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-dvbd9_cc1b5955-9ae6-42a9-8526-a14abb8a9c64/cluster-monitoring-operator/0.log"
Sep 19 09:18:58.798334 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.798289    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-prfk7_66d50cfc-f9e2-4955-9d5d-1214b37b5ba1/monitoring-plugin/0.log"
Sep 19 09:18:58.840447 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.840417    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfskq_734315ee-c009-4d73-8962-b3be234005f7/node-exporter/0.log"
Sep 19 09:18:58.872039 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.872005    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfskq_734315ee-c009-4d73-8962-b3be234005f7/kube-rbac-proxy/0.log"
Sep 19 09:18:58.903366 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:58.903259    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hfskq_734315ee-c009-4d73-8962-b3be234005f7/init-textfile/0.log"
Sep 19 09:18:59.106640 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.106585    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-99f7r_df0eb968-fb92-4a9f-acb1-260b42eddf27/kube-rbac-proxy-main/0.log"
Sep 19 09:18:59.134362 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.134337    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-99f7r_df0eb968-fb92-4a9f-acb1-260b42eddf27/kube-rbac-proxy-self/0.log"
Sep 19 09:18:59.159540 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.159458    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-99f7r_df0eb968-fb92-4a9f-acb1-260b42eddf27/openshift-state-metrics/0.log"
Sep 19 09:18:59.471823 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.471792    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69fcf6478b-r5n97_2e5dba9b-6ee0-4868-b8cc-f01553e9af2d/telemeter-client/0.log"
Sep 19 09:18:59.494553 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.494523    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69fcf6478b-r5n97_2e5dba9b-6ee0-4868-b8cc-f01553e9af2d/reload/0.log"
Sep 19 09:18:59.517236 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:18:59.517206    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69fcf6478b-r5n97_2e5dba9b-6ee0-4868-b8cc-f01553e9af2d/kube-rbac-proxy/0.log"
Sep 19 09:19:01.104975 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:01.104947    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-4tts9_8e683a50-2d02-4b9b-b0cb-8744e2790ef9/networking-console-plugin/0.log"
Sep 19 09:19:01.647029 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:01.647000    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/2.log"
Sep 19 09:19:01.654020 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:01.653967    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-6sw7w_5040f0db-be61-4496-b978-27715767d629/console-operator/3.log"
Sep 19 09:19:02.075349 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:02.075314    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6df54556d4-wrkns_420a2a27-bc57-454f-b61e-c0305b77e891/console/0.log"
Sep 19 09:19:02.116905 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:02.116871    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-45z8n_396b430b-43ea-45c6-9d2a-4c1d83813093/download-server/0.log"
Sep 19 09:19:02.925303 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:02.925268    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"]
Sep 19 09:19:02.932203 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:02.932177    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:02.938569 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:02.938536    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"]
Sep 19 09:19:03.046879 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.046835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-proc\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.047102 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.047009    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87824\" (UniqueName: \"kubernetes.io/projected/aa834596-228a-4cb2-82a6-2aa360726e03-kube-api-access-87824\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.047102 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.047050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-sys\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.047102 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.047080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-lib-modules\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.047295 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.047104    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-podres\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.147621 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87824\" (UniqueName: \"kubernetes.io/projected/aa834596-228a-4cb2-82a6-2aa360726e03-kube-api-access-87824\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-sys\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-lib-modules\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-podres\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-proc\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147784    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-sys\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147862    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-proc\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147963    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-lib-modules\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.148104 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.147963    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa834596-228a-4cb2-82a6-2aa360726e03-podres\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.156583 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.156554    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87824\" (UniqueName: \"kubernetes.io/projected/aa834596-228a-4cb2-82a6-2aa360726e03-kube-api-access-87824\") pod \"perf-node-gather-daemonset-qrfbj\" (UID: \"aa834596-228a-4cb2-82a6-2aa360726e03\") " pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.245452 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.245415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:03.537407 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.537314    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"]
Sep 19 09:19:03.541136 ip-10-0-136-99 kubenswrapper[2584]: W0919 09:19:03.541098    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa834596_228a_4cb2_82a6_2aa360726e03.slice/crio-9b62a925d4a12d72db343a88e225a5ff1da82f560cdd5eb220d91c368924a7b9 WatchSource:0}: Error finding container 9b62a925d4a12d72db343a88e225a5ff1da82f560cdd5eb220d91c368924a7b9: Status 404 returned error can't find the container with id 9b62a925d4a12d72db343a88e225a5ff1da82f560cdd5eb220d91c368924a7b9
Sep 19 09:19:03.543653 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.543593    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vz85_6da27a61-af20-4407-934d-eda26bccb1b6/dns/0.log"
Sep 19 09:19:03.566115 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.566089    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vz85_6da27a61-af20-4407-934d-eda26bccb1b6/kube-rbac-proxy/0.log"
Sep 19 09:19:03.687847 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:03.687822    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2tqn_0be6872e-0257-4d2a-8943-eec60b49bd88/dns-node-resolver/0.log"
Sep 19 09:19:04.050095 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:04.050060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj" event={"ID":"aa834596-228a-4cb2-82a6-2aa360726e03","Type":"ContainerStarted","Data":"a926b48d76043303365520174eb600e9e44472c2af493f3c04e164c604940992"}
Sep 19 09:19:04.050263 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:04.050124    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj" event={"ID":"aa834596-228a-4cb2-82a6-2aa360726e03","Type":"ContainerStarted","Data":"9b62a925d4a12d72db343a88e225a5ff1da82f560cdd5eb220d91c368924a7b9"}
Sep 19 09:19:04.050311 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:04.050276    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:04.068077 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:04.068014    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj" podStartSLOduration=2.067956732 podStartE2EDuration="2.067956732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:19:04.066502549 +0000 UTC m=+3512.151003352" watchObservedRunningTime="2026-09-19 09:19:04.067956732 +0000 UTC m=+3512.152457537"
Sep 19 09:19:04.291664 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:04.291617    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lmzgn_028b379e-a3d7-4094-908c-e51d0ce4d1ff/node-ca/0.log"
Sep 19 09:19:05.079522 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.079494    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9b7b7b666-vbwq7_1211f7b0-61e1-4389-854d-0fef63b6c75f/router/0.log"
Sep 19 09:19:05.465814 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.465783    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmlcm_abb7958d-0540-407c-ba3e-f9d28f8115af/serve-healthcheck-canary/0.log"
Sep 19 09:19:05.824003 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.823894    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-mxmrg_97a94cfc-296d-4617-9970-fc8d7620c8b0/insights-operator/0.log"
Sep 19 09:19:05.825020 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.824969    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-mxmrg_97a94cfc-296d-4617-9970-fc8d7620c8b0/insights-operator/1.log"
Sep 19 09:19:05.917091 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.917036    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drjbp_60649c37-eb40-4171-8c6e-6de49dd4224c/kube-rbac-proxy/0.log"
Sep 19 09:19:05.937461 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.937411    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drjbp_60649c37-eb40-4171-8c6e-6de49dd4224c/exporter/0.log"
Sep 19 09:19:05.958264 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:05.958235    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drjbp_60649c37-eb40-4171-8c6e-6de49dd4224c/extractor/0.log"
Sep 19 09:19:08.455113 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:08.455082    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-6xcvd_0424cbf1-f84f-4ec0-bbff-6095fffc8027/server/0.log"
Sep 19 09:19:08.983172 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:08.983139    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-hdm2c_7f8f9c4b-1f32-4dbd-95dd-5a035308252a/manager/0.log"
Sep 19 09:19:09.181236 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:09.181202    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5bpfd_74811801-cb1a-4466-8962-8d0513ed173a/seaweedfs/0.log"
Sep 19 09:19:09.214771 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:09.214736    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-cjjfk_6224730d-13e7-411b-bfb1-0167fef2fcfc/seaweedfs-tls-custom/0.log"
Sep 19 09:19:09.259138 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:09.259061    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-llzst_1db21908-fbd2-4a6e-8e08-2ed2e313a177/seaweedfs-tls-serving/0.log"
Sep 19 09:19:10.065357 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:10.065329    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-582w2/perf-node-gather-daemonset-qrfbj"
Sep 19 09:19:14.073591 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:14.073554    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jx5qr_7b3384b3-a1de-4edf-898e-dc3a77db41bf/kube-storage-version-migrator-operator/1.log"
Sep 19 09:19:14.077220 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:14.077193    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-jx5qr_7b3384b3-a1de-4edf-898e-dc3a77db41bf/kube-storage-version-migrator-operator/0.log"
Sep 19 09:19:15.146847 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.146781    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5w7k7_bd450284-312e-40a7-9188-8a0692c0b872/kube-multus/0.log"
Sep 19 09:19:15.585554 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.585527    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/kube-multus-additional-cni-plugins/0.log"
Sep 19 09:19:15.607484 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.607456    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/egress-router-binary-copy/0.log"
Sep 19 09:19:15.627562 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.627535    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/cni-plugins/0.log"
Sep 19 09:19:15.648040 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.648011    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/bond-cni-plugin/0.log"
Sep 19 09:19:15.678765 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.678737    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/routeoverride-cni/0.log"
Sep 19 09:19:15.729907 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.729872    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/whereabouts-cni-bincopy/0.log"
Sep 19 09:19:15.748487 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.748455    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r62dt_9920c568-66e7-43c9-839d-8b8846d5d8c2/whereabouts-cni/0.log"
Sep 19 09:19:15.876865 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.876784    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s4btv_d0b18af0-056d-42a2-aee2-e0fb3d43a166/network-metrics-daemon/0.log"
Sep 19 09:19:15.894266 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:15.894238    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s4btv_d0b18af0-056d-42a2-aee2-e0fb3d43a166/kube-rbac-proxy/0.log"
Sep 19 09:19:17.064774 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.064732    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-controller/0.log"
Sep 19 09:19:17.085081 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.085032    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/0.log"
Sep 19 09:19:17.085376 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.085353    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovn-acl-logging/1.log"
Sep 19 09:19:17.112781 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.112725    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/kube-rbac-proxy-node/0.log"
Sep 19 09:19:17.138670 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.138642    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 09:19:17.158015 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.157965    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/northd/0.log"
Sep 19 09:19:17.177829 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.177796    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/nbdb/0.log"
Sep 19 09:19:17.200630 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.200605    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/sbdb/0.log"
Sep 19 09:19:17.269025 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:17.268990    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sbhch_5c2aa2ce-a894-4087-b089-83fa0a5a824c/ovnkube-controller/0.log"
Sep 19 09:19:18.229137 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:18.229105    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-c5sfz_53dccffe-580e-4996-ad9d-0bb39cbeb80f/check-endpoints/0.log"
Sep 19 09:19:18.252377 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:18.252347    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2lpmf_55bd2504-c5a5-4c70-8425-aba0d52bf91e/network-check-target-container/0.log"
Sep 19 09:19:19.220308 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:19.220276    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qgjpr_b50d53d5-1085-4aea-8b4c-bcdcf003d4b5/iptables-alerter/0.log"
Sep 19 09:19:19.871294 ip-10-0-136-99 kubenswrapper[2584]: I0919 09:19:19.871266    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9h2kp_3a2b6c98-8849-4a6f-87a6-648d889e4b4f/tuned/0.log"