Sep 15 16:27:08.493988 ip-10-0-141-27 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 16:27:08.915543 ip-10-0-141-27 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:27:08.915543 ip-10-0-141-27 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 16:27:08.915543 ip-10-0-141-27 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:27:08.915543 ip-10-0-141-27 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 16:27:08.915543 ip-10-0-141-27 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:27:08.916511 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.916429    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 16:27:08.921064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921049    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.921064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921064    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921067    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921071    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921073    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921077    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921080    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921082    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921085    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921089    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921093    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921096    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921098    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921101    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921104    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921106    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921117    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921120    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921122    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921125    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921128    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.921126 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921130    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921133    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921136    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921139    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921142    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921144    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921147    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921150    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921153    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921155    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921158    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921160    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921163    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921165    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921168    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921170    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921172    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921175    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921178    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921180    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.921596 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921183    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921185    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921188    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921190    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921193    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921195    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921197    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921200    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921202    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921205    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921207    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921210    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921212    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921215    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921218    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921220    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921223    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921225    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921228    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921230    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.922104 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921233    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921235    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921238    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921240    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921242    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921245    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921247    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921250    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921252    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921254    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921257    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921260    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921262    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921264    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921267    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921269    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921272    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921274    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921276    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.922575 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921279    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921283    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921286    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921289    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921292    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921296    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921299    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921302    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921305    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921307    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921311    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921314    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921316    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921319    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921321    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921325    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921328    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921331    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.923041 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.921334    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923407    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923416    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923420    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923423    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923426    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923429    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923432    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923435    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923438    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923440    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923443    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923445    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923448    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923450    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923453    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923455    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923458    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923460    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.923481 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923463    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923466    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923468    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923470    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923473    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923475    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923478    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923481    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923484    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923486    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923489    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923492    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923494    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923497    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923499    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923502    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923504    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923507    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923509    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923511    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.923968 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923514    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923516    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923519    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923521    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923524    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923526    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923528    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923531    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923534    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923536    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923539    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923541    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923544    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923547    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923550    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923553    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923555    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923558    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923560    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923563    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.924448 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923565    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923568    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923570    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923572    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923575    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923577    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923580    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923582    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923585    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923587    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923590    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923593    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923596    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923600    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923603    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923605    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923610    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923613    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923616    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.924950 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923619    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923621    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923624    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923626    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923629    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923632    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923634    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923638    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923640    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923643    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923645    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923648    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923650    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923653    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923655    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923658    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923660    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923663    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923665    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923667    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.925406 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923670    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.923672    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924070    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924076    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924079    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924083    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924086    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924089    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924092    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924094    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924097    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924100    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924102    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924105    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924107    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924109    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924112    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924114    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924117    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924121    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.925936 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924123    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924126    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924128    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924131    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924133    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924136    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924139    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924141    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924143    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924146    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924148    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924151    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924154    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924156    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924160    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924163    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924166    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924168    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924171    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924174    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.926411 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924176    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924178    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924182    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924186    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924188    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924191    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924194    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924197    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924199    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924202    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924205    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924208    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924211    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924214    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924216    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924219    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924221    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924223    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924226    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.926912 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924228    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924230    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924233    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924235    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924238    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924240    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924242    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924245    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924247    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924249    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924252    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924254    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924257    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924259    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924262    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924264    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924267    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924269    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924271    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924274    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.927410 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924276    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924278    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924281    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924284    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924288    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924290    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924293    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924295    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924298    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924300    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924303    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924305    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924307    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924310    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924312    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924316    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924319    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924322    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924325    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.927929 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924327    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924329    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.924332    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924401    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924446    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924454    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924459    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924464    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924467    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924471    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924476    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924479    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924482    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924485    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924488    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924491    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924494    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924498    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924500    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924503    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924506    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924510    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924512    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924515    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 16:27:08.928383 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924518    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924522    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924525    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924528    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924531    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924534    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924537    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924540    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924543    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924546    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924548    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924552    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924555    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924558    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924561    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924563    2578 flags.go:64] FLAG: --enable-server="true"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924566    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924570    2578 flags.go:64] FLAG: --event-burst="100"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924574    2578 flags.go:64] FLAG: --event-qps="50"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924576    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924579    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924582    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924585    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924589    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924592    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 16:27:08.929011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924594    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924597    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924600    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924604    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924607    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924610    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924613    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924616    2578 flags.go:64] FLAG: --feature-gates=""
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924619    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924622    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924625    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924628    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924631    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924634    2578 flags.go:64] FLAG: --help="false"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924637    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924640    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924643    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924645    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924649    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924652    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924655    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924657    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924660    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 16:27:08.929599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924663    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924666    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924669    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924672    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924675    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924677    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924680    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924683    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924686    2578 flags.go:64] FLAG: --lock-file=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924689    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924692    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924694    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924700    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924702    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924705    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924708    2578 flags.go:64] FLAG: --logging-format="text"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924711    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924714    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924716    2578 flags.go:64] FLAG: --manifest-url=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924719    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924723    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924726    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924731    2578 flags.go:64] FLAG: --max-pods="110"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924746    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924749    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 16:27:08.930212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924752    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924755    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924758    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924761    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924764    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924777    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924779    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924782    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924786    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924789    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924794    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924797    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924800    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924803    2578 flags.go:64] FLAG: --port="10250"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924806    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924809    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a9ff8347dccf437"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924812    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924814    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924817    2578 flags.go:64] FLAG: --register-node="true"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924820    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924823    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924826    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924828    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924831    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924835    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 16:27:08.930835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924837    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924840    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924843    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924846    2578 flags.go:64] FLAG: --runonce="false"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924849    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924852    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924854    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924857    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924860    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924863    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924865    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924868    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924871    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924874    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924876    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924879    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924882    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924885    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924888    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924893    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924896    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924903    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924906    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924909    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924912    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 16:27:08.932007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924915    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924918    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924921    2578 flags.go:64] FLAG: --v="2"
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924925    2578 flags.go:64] FLAG: --version="false"
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924928    2578 flags.go:64] FLAG: --vmodule=""
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924932    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.924935    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925049    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925054    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925057    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925060    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925063    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925065    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925068    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925070    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925073    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925076    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925078    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925081    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925084    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925086    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.933117 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925089    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925092    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925094    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925097    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925100    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925103    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925105    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925110    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925113    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925116    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925118    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925121    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925123    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925126    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925128    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925131    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925133    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925136    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925138    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925141    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.934064 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925143    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925146    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925148    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925151    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925154    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925156    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925159    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925162    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925164    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925167    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925169    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925173    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925175    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925178    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925182    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925185    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925188    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925191    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.934833 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925195    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925201    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925205    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925208    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925211    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925213    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925216    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925218    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925221    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925223    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925226    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925229    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925232    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925236    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925239    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925242    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925245    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925248    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925250    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925253    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.935280 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925256    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925259    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925261    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925264    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925266    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925269    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925271    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925274    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925276    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925279    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925281    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925284    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925287    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925291    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925293    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925296    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925299    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925301    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925304    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925306    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.935976 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925308    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925311    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925314    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925316    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925319    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925322    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.925324    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.925332    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.933176    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.933193    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.933211    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.933219    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933301    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933310    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933316    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.936460 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933321    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933326    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933330    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933335    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933339    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933344    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933349    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933353    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933358    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933362    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933366    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933370    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933374    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933378    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933382    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933386    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933390    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933394    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933398    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933402    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933406    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.937061 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933411    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933415    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933419    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933423    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933428    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933434    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933438    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933442    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933447    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933451    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933455    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933459    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933463    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933467    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933472    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933476    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933479    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933483    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933488    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933492    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.938029 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933496    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933503    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933509    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933513    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933518    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933522    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933526    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933530    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933535    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933539    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933543    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933547    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933551    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933555    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933559    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933563    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933567    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933572    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933577    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.938673 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933581    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933586    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933590    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933594    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933598    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933602    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933606    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933610    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933614    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933618    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933622    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933626    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933630    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933634    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933638    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933642    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933646    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933650    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933654    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933658    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.939169 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933664    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933670    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933676    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933680    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933685    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933691    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933698    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933703    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933708    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933712    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933717    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933721    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933755    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933760    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933767    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.939717 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933771    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.933780    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933959    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933968    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933976    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933980    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933985    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933989    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.933996    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934001    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934006    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934010    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934015    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934019    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:27:08.940177 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934023    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934027    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934031    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934035    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934040    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934044    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934048    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934052    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934056    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934060    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934064    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934068    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934072    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934075    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934080    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934083    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934088    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934092    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934096    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:27:08.940533 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934100    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934104    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934108    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934112    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934116    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934120    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934124    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934128    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934132    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934135    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934140    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934144    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934148    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934152    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934156    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934160    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934164    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934168    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934172    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934176    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:27:08.941045 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934180    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934184    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934188    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934192    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934197    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934201    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934205    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934209    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934213    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934218    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934223    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934227    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934231    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934235    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934238    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934242    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934246    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934250    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934254    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934258    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:27:08.941552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934262    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934266    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934270    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934274    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934278    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934282    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934286    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934293    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934298    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934303    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934307    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934311    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934315    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934319    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934323    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934327    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934331    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934336    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934340    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934344    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:27:08.942399 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934348    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934352    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934356    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934361    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934365    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934369    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934373    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:08.934377    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.934385    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.934524    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.938270    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.939367    2578 server.go:1019] "Starting client certificate rotation"
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.939469    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:27:08.943367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.940395    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:27:08.943967 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.942251    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:27:08.945820 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.945780    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:27:08.957577 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.957559    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 15 16:27:08.960507 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.960492    2578 log.go:25] "Validated CRI v1 image API"
Sep 15 16:27:08.961379 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.961359    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 16:27:08.967414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.967388    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7dd252eb-ce45-4ee9-99ee-e85a96a6776b:/dev/nvme0n1p4 bb5db811-368c-473c-85b5-caa624ab4ebc:/dev/nvme0n1p3]
Sep 15 16:27:08.967485 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.967413    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 16:27:08.968133 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.968115    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:27:08.973642 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.973530    2578 manager.go:216] Machine: {Timestamp:2026-09-15 16:27:08.971899753 +0000 UTC m=+0.358539013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101354 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212d38e8e4650bac90dc322262b36d SystemUUID:ec212d38-e8e4-650b-ac90-dc322262b36d BootID:05b8e422-0aba-41b2-b4be-a7d467c19f5e Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:58:48:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:58:48:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:ec:f7:04:70:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 16:27:08.973642 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.973634    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 16:27:08.973801 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.973713    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 16:27:08.976831 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.976806    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 16:27:08.976987 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.976834    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 16:27:08.978142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.978132    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 16:27:08.978179 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.978146    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 16:27:08.978179 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.978153    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:27:08.979314 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.979304    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:27:08.980078 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.980069    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 16:27:08.984258 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984248    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:27:08.984368 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984359    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 16:27:08.984403 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984388    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 16:27:08.984403 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984398    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 16:27:08.984457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984411    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 16:27:08.984457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984421    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 15 16:27:08.984457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984445    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 16:27:08.984821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984805    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:27:08.984821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.984825    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:27:08.987102 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.987083    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 16:27:08.989357 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.989339    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 16:27:08.989451 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.989371    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 16:27:08.991850 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991834    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 16:27:08.991850 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991853    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991860    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991866    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991874    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991880    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991893    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991898    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991904    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991909    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991918    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 16:27:08.991993 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.991923    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 16:27:08.992666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.992657    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 16:27:08.992666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.992665    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 16:27:08.995072 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:08.995050    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 16:27:08.995203 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.995186    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 16:27:08.995259 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:08.995192    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 16:27:08.996280 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.996268    2578 server.go:1268] "Started kubelet"
Sep 15 16:27:08.996368 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.996333    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 16:27:08.996466 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.996348    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 16:27:08.996466 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:08.996428    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 16:27:09.000965 ip-10-0-141-27 systemd[1]: Started Kubernetes Kubelet.
Sep 15 16:27:09.001666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.001592    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 16:27:09.002835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.002819    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 16:27:09.002983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.002964    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bj9w"
Sep 15 16:27:09.005225 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.005203    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 16:27:09.008120 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.008106    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bj9w"
Sep 15 16:27:09.008344 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.007021    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-27.ec2.internal.18d58b56219107c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-27.ec2.internal,UID:ip-10-0-141-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-27.ec2.internal,},FirstTimestamp:2026-09-15 16:27:08.99624135 +0000 UTC m=+0.382880664,LastTimestamp:2026-09-15 16:27:08.99624135 +0000 UTC m=+0.382880664,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-27.ec2.internal,}"
Sep 15 16:27:09.009870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.009852    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 16:27:09.009870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.009856    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 16:27:09.010074 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.009921    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.010457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010444    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 16:27:09.010556 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010503    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 16:27:09.010642 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010620    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 16:27:09.010920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010781    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 16:27:09.010978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010923    2578 factory.go:55] Registering systemd factory
Sep 15 16:27:09.010978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010934    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 15 16:27:09.011070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010990    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 16:27:09.011070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.010997    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 16:27:09.011236 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.011205    2578 factory.go:153] Registering CRI-O factory
Sep 15 16:27:09.011236 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.011227    2578 factory.go:223] Registration of the crio container factory successfully
Sep 15 16:27:09.011355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.011246    2578 factory.go:103] Registering Raw factory
Sep 15 16:27:09.011355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.011260    2578 manager.go:1196] Started watching for new ooms in manager
Sep 15 16:27:09.011596 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.011583    2578 manager.go:349] Starting recovery of all containers
Sep 15 16:27:09.014942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.014921    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:09.019391 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.019373    2578 manager.go:354] Recovery completed
Sep 15 16:27:09.020504 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.020481    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-27.ec2.internal\" not found" node="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.025348 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.025336    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.030329 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.030314    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.030389 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.030342    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.030389 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.030358    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.032180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032167    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 16:27:09.032180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032176    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.032180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032180    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 16:27:09.032350 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032199    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:27:09.032953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032938    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.033006 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032964    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.033006 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.032973    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.033537 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.033525    2578 policy_none.go:49] "None policy: Start"
Sep 15 16:27:09.033588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.033544    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 16:27:09.033588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.033557    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 16:27:09.034988 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.034975    2578 policy_none.go:47] "Start"
Sep 15 16:27:09.079408 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.079383    2578 manager.go:337] "Starting Device Plugin manager"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.079433    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.079448    2578 server.go:87] "Starting device plugin registration server"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.079885    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.079902    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.080028    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.080107    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.080117    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.080634    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 16:27:09.082793 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.080688    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.147035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.146992    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 16:27:09.148364 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.148342    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 16:27:09.148461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.148395    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 16:27:09.148461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.148420    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 16:27:09.148568 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.148477    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 16:27:09.150548 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.150529    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:09.180775 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.180667    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.181503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.181489    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.181579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.181520    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.181579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.181535    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.181579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.181565    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.189645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.189625    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.189718 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.189646    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-27.ec2.internal\": node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.206615 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.206589    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.249579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.249537    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"]
Sep 15 16:27:09.249631 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.249622    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.251056 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.251043    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.251123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.251069    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.251123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.251079    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.252346 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.252334    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.252475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.252461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.252536 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.252495    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.253059 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253040    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.253140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253040    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.253140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253071    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.253140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253086    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.253140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253087    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.253140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.253104    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.254485 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.254468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.254526 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.254495    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:27:09.255174 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.255157    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:27:09.255264 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.255191    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:27:09.255264 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.255203    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:27:09.277263 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.277237    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-27.ec2.internal\" not found" node="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.281478 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.281461    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-27.ec2.internal\" not found" node="ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.307661 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.307639    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.312132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.312112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d09620bd92a67858915e7d86ee2bdcad-config\") pod \"kube-apiserver-proxy-ip-10-0-141-27.ec2.internal\" (UID: \"d09620bd92a67858915e7d86ee2bdcad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.312206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.312144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.312206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.312174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.408378 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.408355    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.412696 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.412797 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.412859 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.412859 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d09620bd92a67858915e7d86ee2bdcad-config\") pod \"kube-apiserver-proxy-ip-10-0-141-27.ec2.internal\" (UID: \"d09620bd92a67858915e7d86ee2bdcad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.412859 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab8bd5fea60f6c83158110ca739ec8e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal\" (UID: \"ab8bd5fea60f6c83158110ca739ec8e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.412859 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.412856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d09620bd92a67858915e7d86ee2bdcad-config\") pod \"kube-apiserver-proxy-ip-10-0-141-27.ec2.internal\" (UID: \"d09620bd92a67858915e7d86ee2bdcad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.509172 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.509104    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.579522 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.579490    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.584662 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.584647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:09.609349 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.609321    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.709927 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.709896    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.810388 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.810320    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.910837 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:09.910805    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:09.939158 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.939132    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 16:27:09.939545 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.939271    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:27:09.939545 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:09.939287    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:27:10.010905 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.010841    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 16:22:09 +0000 UTC" deadline="2028-03-24 02:48:15.139923429 +0000 UTC"
Sep 15 16:27:10.010905 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:10.010863    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-27.ec2.internal\" not found"
Sep 15 16:27:10.010905 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.010872    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13330h21m5.129054415s"
Sep 15 16:27:10.055149 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:10.055114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8bd5fea60f6c83158110ca739ec8e0.slice/crio-71e0bf553a76d79f298b98f6680b7356ba7022a9d0cb30e6eea53446cdff2168 WatchSource:0}: Error finding container 71e0bf553a76d79f298b98f6680b7356ba7022a9d0cb30e6eea53446cdff2168: Status 404 returned error can't find the container with id 71e0bf553a76d79f298b98f6680b7356ba7022a9d0cb30e6eea53446cdff2168
Sep 15 16:27:10.059009 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.058991    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:10.059323 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.059310    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:27:10.110823 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.110793    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:10.122257 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.122237    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:27:10.123975 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.123963    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal"
Sep 15 16:27:10.133974 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.133957    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:27:10.152360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.152319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal" event={"ID":"ab8bd5fea60f6c83158110ca739ec8e0","Type":"ContainerStarted","Data":"71e0bf553a76d79f298b98f6680b7356ba7022a9d0cb30e6eea53446cdff2168"}
Sep 15 16:27:10.206483 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:10.206452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09620bd92a67858915e7d86ee2bdcad.slice/crio-bf0f318cbb6c528273234a2f3ba39bb9b410caff8583b8c92a5da0fc09db71d1 WatchSource:0}: Error finding container bf0f318cbb6c528273234a2f3ba39bb9b410caff8583b8c92a5da0fc09db71d1: Status 404 returned error can't find the container with id bf0f318cbb6c528273234a2f3ba39bb9b410caff8583b8c92a5da0fc09db71d1
Sep 15 16:27:10.555196 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.555166    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:10.824068 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.823988    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:10.985938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.985903    2578 apiserver.go:52] "Watching apiserver"
Sep 15 16:27:10.992091 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.992065    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 16:27:10.992530 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.992506    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552","openshift-cluster-node-tuning-operator/tuned-g28k2","openshift-dns/node-resolver-kwhm5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal","openshift-multus/multus-xsxqb","openshift-network-operator/iptables-alerter-d52kx","openshift-ovn-kubernetes/ovnkube-node-qtgkt","kube-system/global-pull-secret-syncer-tpn5j","kube-system/konnectivity-agent-9snvm","kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal","openshift-image-registry/node-ca-whwt5","openshift-multus/multus-additional-cni-plugins-4nsv5","openshift-multus/network-metrics-daemon-sjswr","openshift-network-diagnostics/network-check-target-q5g8m"]
Sep 15 16:27:10.994627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.994601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:10.994726 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:10.994693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:10.997051 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.997024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.000167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.999457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.000167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:10.999768    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.000167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.000001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jlxx6\""
Sep 15 16:27:11.006856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.005509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.006856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.006130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.009627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.009604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.009910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.009773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ckgm5\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010206    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bbdlx\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010254    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010307    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.010421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010389    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 16:27:11.010765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.010619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.012070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.012049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.013040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.012721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzqqp\""
Sep 15 16:27:11.014025 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.014010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.014268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.014365    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5f6c\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.016941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-hostroot\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-conf\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-lib-modules\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-tmp\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-sys-fs\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cnibin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-netns\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-kubelet\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-modprobe-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-systemd\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-etc-tuned\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.020833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-socket-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-registration-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-etc-selinux\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cni-binary-copy\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-conf-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-daemon-config\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-etc-kubernetes\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p527\" (UniqueName: \"kubernetes.io/projected/54a962b6-0a66-47b4-8205-4bd659c5c8e0-kube-api-access-4p527\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-device-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzcx\" (UniqueName: \"kubernetes.io/projected/b15f4535-2258-4631-9616-f79019a4f309-kube-api-access-tvzcx\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-kubernetes\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-sys\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-var-lib-kubelet\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2p4\" (UniqueName: \"kubernetes.io/projected/fa5880e1-7205-4d98-b561-c00b65624313-kube-api-access-jz2p4\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.020988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-socket-dir-parent\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.021966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-host\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-bin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-multus\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-multus-certs\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a962b6-0a66-47b4-8205-4bd659c5c8e0-iptables-alerter-script\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a962b6-0a66-47b4-8205-4bd659c5c8e0-host-slash\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34b7f025-99b9-4a22-a786-de64866c0071-hosts-file\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b7f025-99b9-4a22-a786-de64866c0071-tmp-dir\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhhm\" (UniqueName: \"kubernetes.io/projected/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kube-api-access-5dhhm\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysconfig\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-system-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-k8s-cni-cncf-io\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsklh\" (UniqueName: \"kubernetes.io/projected/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-kube-api-access-lsklh\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwts2\" (UniqueName: \"kubernetes.io/projected/34b7f025-99b9-4a22-a786-de64866c0071-kube-api-access-rwts2\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.022756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.021500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-run\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.025222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.025500    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wc26n\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.025729    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.026011    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.026356    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.026591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.026904    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.028310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.029097 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.028659    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:11.029610 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.029166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.030387 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.030368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:11.030869 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.030761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:11.031280 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.031105    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:27:11.032325 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.032303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mtkxk\""
Sep 15 16:27:11.032580 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.032403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 16:27:11.032693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.032675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 16:27:11.032844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.032824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.035345 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.035325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 16:27:11.036942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.036254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 16:27:11.036942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.036462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wqb6z\""
Sep 15 16:27:11.036942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.036721    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 16:27:11.037978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.037551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.040706 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.040688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 16:27:11.040858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.040688    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 16:27:11.041389 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.041370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnnlg\""
Sep 15 16:27:11.065791 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.065761    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wfrr9"
Sep 15 16:27:11.075167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.075143    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wfrr9"
Sep 15 16:27:11.112254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.112227    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 16:27:11.122517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-tmp\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.122641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-sys-fs\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.122641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-netns\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.122641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-kubelet\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.122641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-modprobe-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.122641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-systemd\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-etc-tuned\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-socket-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-etc-selinux\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-systemd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-node-log\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.122913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-conf-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p527\" (UniqueName: \"kubernetes.io/projected/54a962b6-0a66-47b4-8205-4bd659c5c8e0-kube-api-access-4p527\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.122993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-script-lib\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2465ff10-35ea-4eb1-8440-c63a317977a7-konnectivity-ca\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/932e1943-4d3d-42d7-bab6-7e4270936ec4-serviceca\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccclf\" (UniqueName: \"kubernetes.io/projected/932e1943-4d3d-42d7-bab6-7e4270936ec4-kube-api-access-ccclf\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-device-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-sys\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-var-lib-kubelet\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-socket-dir-parent\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-netns\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123299    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-ovn\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-host\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-netns\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-bin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-kubelet\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-kubelet\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b7f025-99b9-4a22-a786-de64866c0071-tmp-dir\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhhm\" (UniqueName: \"kubernetes.io/projected/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kube-api-access-5dhhm\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.123569    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysconfig\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-system-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.123668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:11.623625977 +0000 UTC m=+3.010265244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-conf-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsklh\" (UniqueName: \"kubernetes.io/projected/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-kube-api-access-lsklh\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.123762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-system-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-dbus\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-sys\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-sys-fs\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-modprobe-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-systemd\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.123947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-host\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-bin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-socket-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-etc-selinux\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.124534 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysconfig\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b7f025-99b9-4a22-a786-de64866c0071-tmp-dir\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-device-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-socket-dir-parent\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-cni-dir\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwts2\" (UniqueName: \"kubernetes.io/projected/34b7f025-99b9-4a22-a786-de64866c0071-kube-api-access-rwts2\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-var-lib-kubelet\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-run\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-hostroot\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-run\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.124995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-hostroot\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48lv\" (UniqueName: \"kubernetes.io/projected/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-kube-api-access-p48lv\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-conf\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-lib-modules\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cnibin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-conf\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-lib-modules\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2465ff10-35ea-4eb1-8440-c63a317977a7-agent-certs\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cnibin\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-registration-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cni-binary-copy\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19193dc6-7d7e-4ecb-affa-a9879ec38e90-registration-dir\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-slash\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-kubelet-config\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-daemon-config\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-etc-kubernetes\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.125970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-bin\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzcx\" (UniqueName: \"kubernetes.io/projected/b15f4535-2258-4631-9616-f79019a4f309-kube-api-access-tvzcx\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-kubernetes\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2p4\" (UniqueName: \"kubernetes.io/projected/fa5880e1-7205-4d98-b561-c00b65624313-kube-api-access-jz2p4\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-multus\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-multus-certs\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhkw\" (UniqueName: \"kubernetes.io/projected/94305cb9-6f3f-4231-9211-c4bac29600f7-kube-api-access-hrhkw\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-systemd-units\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-log-socket\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.125973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a962b6-0a66-47b4-8205-4bd659c5c8e0-iptables-alerter-script\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a962b6-0a66-47b4-8205-4bd659c5c8e0-host-slash\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-cni-binary-copy\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-etc-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/932e1943-4d3d-42d7-bab6-7e4270936ec4-host\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.126860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34b7f025-99b9-4a22-a786-de64866c0071-hosts-file\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-multus-daemon-config\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-system-cni-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-etc-kubernetes\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-cnibin\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-k8s-cni-cncf-io\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-var-lib-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-netd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-multus-certs\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-config\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-env-overrides\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovn-node-metrics-cert\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34b7f025-99b9-4a22-a786-de64866c0071-hosts-file\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-var-lib-cni-multus\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54a962b6-0a66-47b4-8205-4bd659c5c8e0-iptables-alerter-script\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54a962b6-0a66-47b4-8205-4bd659c5c8e0-host-slash\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.127735 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-sysctl-d\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.128609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-host-run-k8s-cni-cncf-io\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.128609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.126831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa5880e1-7205-4d98-b561-c00b65624313-etc-kubernetes\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.128609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.127714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-etc-tuned\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.128609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.128398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa5880e1-7205-4d98-b561-c00b65624313-tmp\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.142113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.142085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p527\" (UniqueName: \"kubernetes.io/projected/54a962b6-0a66-47b4-8205-4bd659c5c8e0-kube-api-access-4p527\") pod \"iptables-alerter-d52kx\" (UID: \"54a962b6-0a66-47b4-8205-4bd659c5c8e0\") " pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.149145 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.149110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzcx\" (UniqueName: \"kubernetes.io/projected/b15f4535-2258-4631-9616-f79019a4f309-kube-api-access-tvzcx\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.149239 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.149207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhhm\" (UniqueName: \"kubernetes.io/projected/19193dc6-7d7e-4ecb-affa-a9879ec38e90-kube-api-access-5dhhm\") pod \"aws-ebs-csi-driver-node-cz552\" (UID: \"19193dc6-7d7e-4ecb-affa-a9879ec38e90\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.149395 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.149358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwts2\" (UniqueName: \"kubernetes.io/projected/34b7f025-99b9-4a22-a786-de64866c0071-kube-api-access-rwts2\") pod \"node-resolver-kwhm5\" (UID: \"34b7f025-99b9-4a22-a786-de64866c0071\") " pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.150768 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.150736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsklh\" (UniqueName: \"kubernetes.io/projected/24ee9eeb-a6e5-4207-b5a1-a84b427c76bd-kube-api-access-lsklh\") pod \"multus-xsxqb\" (UID: \"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd\") " pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.151526 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.151494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2p4\" (UniqueName: \"kubernetes.io/projected/fa5880e1-7205-4d98-b561-c00b65624313-kube-api-access-jz2p4\") pod \"tuned-g28k2\" (UID: \"fa5880e1-7205-4d98-b561-c00b65624313\") " pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.155316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.155290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal" event={"ID":"d09620bd92a67858915e7d86ee2bdcad","Type":"ContainerStarted","Data":"bf0f318cbb6c528273234a2f3ba39bb9b410caff8583b8c92a5da0fc09db71d1"}
Sep 15 16:27:11.227938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.227898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-systemd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.227938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.227932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228150 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.227960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-node-log\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228454 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.228576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-script-lib\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2465ff10-35ea-4eb1-8440-c63a317977a7-konnectivity-ca\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.228704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/932e1943-4d3d-42d7-bab6-7e4270936ec4-serviceca\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.228704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccclf\" (UniqueName: \"kubernetes.io/projected/932e1943-4d3d-42d7-bab6-7e4270936ec4-kube-api-access-ccclf\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.228704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-netns\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-ovn\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-systemd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.228944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-kubelet\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.228977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-kubelet\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.229106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-dbus\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.229316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.229316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48lv\" (UniqueName: \"kubernetes.io/projected/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-kube-api-access-p48lv\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229453 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2465ff10-35ea-4eb1-8440-c63a317977a7-agent-certs\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.229453 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-slash\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-kubelet-config\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.229518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.229583 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-bin\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229583 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.229643 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhkw\" (UniqueName: \"kubernetes.io/projected/94305cb9-6f3f-4231-9211-c4bac29600f7-kube-api-access-hrhkw\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.229680 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-systemd-units\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229715 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-log-socket\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229746 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-etc-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229777 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.229838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/932e1943-4d3d-42d7-bab6-7e4270936ec4-host\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.229899 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-system-cni-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.229899 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-cnibin\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.230010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-var-lib-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.230010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-netd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.230010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.229991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-config\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.230107 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.230017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-env-overrides\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.230107 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.230050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovn-node-metrics-cert\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.230107 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.230082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:11.230613 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.230592    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.231210 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.231189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-script-lib\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.231281 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.231266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-node-log\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.231821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.231788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-binary-copy\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.231953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.231935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-etc-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232023 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232023 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.231954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-systemd-units\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-var-lib-openvswitch\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232145 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-log-socket\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-netd\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-dbus\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.232289 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-system-cni-dir\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.232289 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-run-ovn\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-cni-bin\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.232390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/932e1943-4d3d-42d7-bab6-7e4270936ec4-host\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.232499 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.232392    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:11.232499 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.232456    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:11.732437848 +0000 UTC m=+3.119077111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:11.232619 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94305cb9-6f3f-4231-9211-c4bac29600f7-cnibin\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.232787 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.232759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-env-overrides\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.233690 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/932e1943-4d3d-42d7-bab6-7e4270936ec4-serviceca\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.233773 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.233866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.233944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-run-netns\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.234029 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/94305cb9-6f3f-4231-9211-c4bac29600f7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.234029 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.233981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a16f1e9a-a3bb-450f-881a-15e066a3b639-kubelet-config\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.234440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.234370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-host-slash\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.234546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.234441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2465ff10-35ea-4eb1-8440-c63a317977a7-konnectivity-ca\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.235434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.235401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovnkube-config\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.235886 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.235861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-ovn-node-metrics-cert\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.236529 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.236496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2465ff10-35ea-4eb1-8440-c63a317977a7-agent-certs\") pod \"konnectivity-agent-9snvm\" (UID: \"2465ff10-35ea-4eb1-8440-c63a317977a7\") " pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.243359 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.243341    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:11.243359 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.243362    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:11.243505 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.243373    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:11.243505 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.243429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:11.743410436 +0000 UTC m=+3.130049685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:11.246996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.246966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccclf\" (UniqueName: \"kubernetes.io/projected/932e1943-4d3d-42d7-bab6-7e4270936ec4-kube-api-access-ccclf\") pod \"node-ca-whwt5\" (UID: \"932e1943-4d3d-42d7-bab6-7e4270936ec4\") " pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.249096 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.249070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48lv\" (UniqueName: \"kubernetes.io/projected/a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590-kube-api-access-p48lv\") pod \"ovnkube-node-qtgkt\" (UID: \"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.250407 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.250386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhkw\" (UniqueName: \"kubernetes.io/projected/94305cb9-6f3f-4231-9211-c4bac29600f7-kube-api-access-hrhkw\") pod \"multus-additional-cni-plugins-4nsv5\" (UID: \"94305cb9-6f3f-4231-9211-c4bac29600f7\") " pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.311843 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.311806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g28k2"
Sep 15 16:27:11.328113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.328053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552"
Sep 15 16:27:11.338588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.338565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kwhm5"
Sep 15 16:27:11.342610 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.342579    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:27:11.346325 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.346305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xsxqb"
Sep 15 16:27:11.355887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.355867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d52kx"
Sep 15 16:27:11.363796 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.363778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:11.370517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.370482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:11.378263 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.378240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-whwt5"
Sep 15 16:27:11.384044 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.384023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4nsv5"
Sep 15 16:27:11.633815 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.633695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:11.634005 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.633887    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:11.634005 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.633972    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:12.633952249 +0000 UTC m=+4.020591505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:11.734662 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.734625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:11.734864 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.734818    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:11.734930 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.734886    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:12.734867502 +0000 UTC m=+4.121506756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:11.835940 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:11.835895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:11.836110 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.836081    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:11.836157 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.836110    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:11.836157 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.836126    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:11.836239 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:11.836193    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:12.836176739 +0000 UTC m=+4.222815992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:12.078153 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:12.078047    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:22:11 +0000 UTC" deadline="2028-03-24 04:37:48.188441417 +0000 UTC"
Sep 15 16:27:12.078153 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:12.078090    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13332h10m36.110356863s"
Sep 15 16:27:12.204539 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:12.204356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2465ff10_35ea_4eb1_8440_c63a317977a7.slice/crio-a2632c46b9ad409b04ba68ee8fc20445aa88843e9b2da1217c8afb96bae35ef0 WatchSource:0}: Error finding container a2632c46b9ad409b04ba68ee8fc20445aa88843e9b2da1217c8afb96bae35ef0: Status 404 returned error can't find the container with id a2632c46b9ad409b04ba68ee8fc20445aa88843e9b2da1217c8afb96bae35ef0
Sep 15 16:27:12.207405 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:12.207381    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19193dc6_7d7e_4ecb_affa_a9879ec38e90.slice/crio-0b109f984aa5533c112add73371ae885d43ac288afc4d9cc11ffb2855ac0344e WatchSource:0}: Error finding container 0b109f984aa5533c112add73371ae885d43ac288afc4d9cc11ffb2855ac0344e: Status 404 returned error can't find the container with id 0b109f984aa5533c112add73371ae885d43ac288afc4d9cc11ffb2855ac0344e
Sep 15 16:27:12.212027 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:12.211496    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b7f025_99b9_4a22_a786_de64866c0071.slice/crio-dfae4923af5fa88aada50e4f83e4c44e5d15018a2c811b225bb56ff6873700d1 WatchSource:0}: Error finding container dfae4923af5fa88aada50e4f83e4c44e5d15018a2c811b225bb56ff6873700d1: Status 404 returned error can't find the container with id dfae4923af5fa88aada50e4f83e4c44e5d15018a2c811b225bb56ff6873700d1
Sep 15 16:27:12.212479 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:12.212458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a962b6_0a66_47b4_8205_4bd659c5c8e0.slice/crio-1f56d09b1b7114233ad573320a01ea99120b88e2963f0471560adc13021e0d81 WatchSource:0}: Error finding container 1f56d09b1b7114233ad573320a01ea99120b88e2963f0471560adc13021e0d81: Status 404 returned error can't find the container with id 1f56d09b1b7114233ad573320a01ea99120b88e2963f0471560adc13021e0d81
Sep 15 16:27:12.214667 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:27:12.214207    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5880e1_7205_4d98_b561_c00b65624313.slice/crio-bba850ea2e743a5ad6c6fbf47b5297e0e016e8f8a9ec90029b53df3a20109160 WatchSource:0}: Error finding container bba850ea2e743a5ad6c6fbf47b5297e0e016e8f8a9ec90029b53df3a20109160: Status 404 returned error can't find the container with id bba850ea2e743a5ad6c6fbf47b5297e0e016e8f8a9ec90029b53df3a20109160
Sep 15 16:27:12.642816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:12.642771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:12.642986 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.642934    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:12.643030 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.642993    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:14.642977901 +0000 UTC m=+6.029617153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:12.743829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:12.743739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:12.744007 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.743981    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:12.744134 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.744060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:14.744038636 +0000 UTC m=+6.130677897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:12.845913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:12.845196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:12.845913 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.845405    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:12.845913 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.845431    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:12.845913 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.845445    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:12.845913 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:12.845518    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:14.845496024 +0000 UTC m=+6.232135294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:13.079007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.078896    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:22:11 +0000 UTC" deadline="2028-05-23 19:11:14.413167372 +0000 UTC"
Sep 15 16:27:13.079007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.078957    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14786h44m1.334214284s"
Sep 15 16:27:13.151688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.151656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:13.151862 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:13.151772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:13.152223 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.152196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:13.152342 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:13.152318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:13.152410 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.152395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:13.152483 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:13.152462    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:13.165271 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.165235    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab8bd5fea60f6c83158110ca739ec8e0" containerID="4f84feb34faee35195c84acb56c746a2dc11997396cbbf35c10a690777fdbfa6" exitCode=0
Sep 15 16:27:13.165662 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.165608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal" event={"ID":"ab8bd5fea60f6c83158110ca739ec8e0","Type":"ContainerDied","Data":"4f84feb34faee35195c84acb56c746a2dc11997396cbbf35c10a690777fdbfa6"}
Sep 15 16:27:13.172838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.172710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"2ddfacc369ecdf7f82b44616aa6f61e0f8c7212aa4eebf47a80a5e9b0a6add28"}
Sep 15 16:27:13.177128 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.176166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whwt5" event={"ID":"932e1943-4d3d-42d7-bab6-7e4270936ec4","Type":"ContainerStarted","Data":"add8e738e3119b9ff23889ec14b0d309bec436843cc87d0fb9fc13b2249d9b70"}
Sep 15 16:27:13.178228 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.178201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552" event={"ID":"19193dc6-7d7e-4ecb-affa-a9879ec38e90","Type":"ContainerStarted","Data":"0b109f984aa5533c112add73371ae885d43ac288afc4d9cc11ffb2855ac0344e"}
Sep 15 16:27:13.185311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.185219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9snvm" event={"ID":"2465ff10-35ea-4eb1-8440-c63a317977a7","Type":"ContainerStarted","Data":"a2632c46b9ad409b04ba68ee8fc20445aa88843e9b2da1217c8afb96bae35ef0"}
Sep 15 16:27:13.190980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.190924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal" event={"ID":"d09620bd92a67858915e7d86ee2bdcad","Type":"ContainerStarted","Data":"78b782388ee09f71a126a1040f747d89e1ab4e06d11dbc934dede2eee451f808"}
Sep 15 16:27:13.200949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.200920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xsxqb" event={"ID":"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd","Type":"ContainerStarted","Data":"8ed1116630438e5b6da32bc5127616dc951fc0fed2bed91958555b5a4ab5f994"}
Sep 15 16:27:13.202569 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.202545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerStarted","Data":"0c079c25ccb4e6d6fa5863077053f1eec8c693b8ef9fdd26ff32597532e7e63f"}
Sep 15 16:27:13.204468 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.204436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g28k2" event={"ID":"fa5880e1-7205-4d98-b561-c00b65624313","Type":"ContainerStarted","Data":"bba850ea2e743a5ad6c6fbf47b5297e0e016e8f8a9ec90029b53df3a20109160"}
Sep 15 16:27:13.208802 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.208736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d52kx" event={"ID":"54a962b6-0a66-47b4-8205-4bd659c5c8e0","Type":"ContainerStarted","Data":"1f56d09b1b7114233ad573320a01ea99120b88e2963f0471560adc13021e0d81"}
Sep 15 16:27:13.213471 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:13.213447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kwhm5" event={"ID":"34b7f025-99b9-4a22-a786-de64866c0071","Type":"ContainerStarted","Data":"dfae4923af5fa88aada50e4f83e4c44e5d15018a2c811b225bb56ff6873700d1"}
Sep 15 16:27:14.222911 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:14.222873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal" event={"ID":"ab8bd5fea60f6c83158110ca739ec8e0","Type":"ContainerStarted","Data":"b773988a557ea0435e6acf327923546f31a70578d87df3e5856a45dd0b03bd52"}
Sep 15 16:27:14.246366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:14.245446    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-27.ec2.internal" podStartSLOduration=4.245422613 podStartE2EDuration="4.245422613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:13.20854525 +0000 UTC m=+4.595184522" watchObservedRunningTime="2026-09-15 16:27:14.245422613 +0000 UTC m=+5.632061886"
Sep 15 16:27:14.665881 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:14.665839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:14.666086 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.666041    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:14.666156 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.666106    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.666087162 +0000 UTC m=+10.052726426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:14.766280 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:14.766208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:14.766466 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.766411    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:14.766536 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.766478    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.766458851 +0000 UTC m=+10.153098113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:14.867094 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:14.867053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:14.867278 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.867239    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:14.867278 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.867263    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:14.867278 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.867275    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:14.867445 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:14.867327    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.867312729 +0000 UTC m=+10.253951980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:15.150552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:15.149934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:15.150552 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:15.150058    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:15.150552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:15.150421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:15.150552 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:15.150509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:15.150881 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:15.150706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:15.150934 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:15.150876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:17.151008 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:17.150938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:17.151518 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:17.151088    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:17.151518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:17.151501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:17.151518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:17.151510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:17.151680 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:17.151609    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:17.151846 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:17.151739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:18.702004 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:18.701959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:18.702480 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.702158    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:18.702480 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.702228    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:26.702207865 +0000 UTC m=+18.088847115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:18.803312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:18.803274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:18.803648 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.803625    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:18.803719 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.803707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:26.803686048 +0000 UTC m=+18.190325311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:18.904339 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:18.904301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:18.904517 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.904483    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:18.904517 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.904506    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:18.904517 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.904519    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:18.904664 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:18.904565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:26.904552775 +0000 UTC m=+18.291192033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:19.149794 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:19.149759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:19.149956 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:19.149859    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:19.149956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:19.149908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:19.150233 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:19.150045    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:19.150233 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:19.150070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:19.150233 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:19.150188    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:21.149378 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:21.149342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:21.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:21.149485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:21.149858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:21.149341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:21.149858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:21.149522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:21.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:21.149582    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:21.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:21.149650    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:23.149109 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:23.148904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:23.149503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:23.148916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:23.149503 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:23.149194    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:23.149503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:23.148972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:23.149503 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:23.149256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:23.149503 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:23.149348    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:25.149527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:25.149495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:25.149984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:25.149498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:25.149984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:25.149603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:25.149984 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:25.149610    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:25.149984 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:25.149686    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:25.149984 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:25.149777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:26.761460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:26.761418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:26.761893 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.761606    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:26.761893 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.761688    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.761665902 +0000 UTC m=+34.148305165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:26.862586 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:26.862542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:26.862779 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.862707    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:26.862859 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.862801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.862781459 +0000 UTC m=+34.249420704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:26.962918 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:26.962887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:26.963099 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.963024    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:26.963099 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.963042    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:26.963099 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.963050    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:26.963099 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:26.963099    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.963086647 +0000 UTC m=+34.349725891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:27.149584 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:27.149549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:27.149584 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:27.149571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:27.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:27.149671    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:27.149858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:27.149767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:27.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:27.149823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:27.149858 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:27.149756    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:29.150233 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:29.150194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:29.150697 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:29.150292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:29.150697 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:29.150310    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:29.150697 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:29.150414    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:29.150697 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:29.150448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:29.150697 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:29.150527    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:31.149880 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.149613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:31.150775 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.149622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:31.150775 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:31.149983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:31.150775 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.149627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:31.150775 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:31.150034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:31.150775 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:31.150103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:31.261523 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"57a7227220945c52d976b3b46963db77d94115b1710af92d3ab00bdf6ab166e8"}
Sep 15 16:27:31.261668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"eb84cb778b424509a9a2aabd453501eae929c79c87f995c27d445b422ac7744b"}
Sep 15 16:27:31.261668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"393bcfb7c81c284abcee74079d19e29874d1d56e84c38eef38811c5f8dc984f0"}
Sep 15 16:27:31.261668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"dcfad577930c604cc649ddc81b8d1e2067585f402dda49051520dfce2144ed41"}
Sep 15 16:27:31.261816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"d16d058da792a1261d65cf4dada4cb812b196928cb940a2ab8895a92c0f2f58a"}
Sep 15 16:27:31.261816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.261753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"bdf27752fe77b7749fa8baabedbbeb7e8b4123e7afe47f53745689fc074a7ce1"}
Sep 15 16:27:31.262970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.262950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-whwt5" event={"ID":"932e1943-4d3d-42d7-bab6-7e4270936ec4","Type":"ContainerStarted","Data":"48d16b07cc6829ae7d471d535a1c0b3eee1a5f622cadee4c7df66da6d7081d29"}
Sep 15 16:27:31.264453 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.264430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552" event={"ID":"19193dc6-7d7e-4ecb-affa-a9879ec38e90","Type":"ContainerStarted","Data":"7d84a548138cd8d5b48909e2f22fdf6d8919a40b62bc41a72f2e56a7ff42fa98"}
Sep 15 16:27:31.265998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.265964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9snvm" event={"ID":"2465ff10-35ea-4eb1-8440-c63a317977a7","Type":"ContainerStarted","Data":"9d2822ddf8b4ca6c648137cde6499a8ccead8dfde2790a74759a627689a44e76"}
Sep 15 16:27:31.267445 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.267418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xsxqb" event={"ID":"24ee9eeb-a6e5-4207-b5a1-a84b427c76bd","Type":"ContainerStarted","Data":"e23864e8badc22f548fc0fa2a1e2242a2bc92323e706d8cbf438efc1133a4314"}
Sep 15 16:27:31.268854 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.268833    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="22563701a97bfeddf2cc53bb116b771dd9ab724935da468383540f47f527bb40" exitCode=0
Sep 15 16:27:31.268958 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.268891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"22563701a97bfeddf2cc53bb116b771dd9ab724935da468383540f47f527bb40"}
Sep 15 16:27:31.270296 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.270243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g28k2" event={"ID":"fa5880e1-7205-4d98-b561-c00b65624313","Type":"ContainerStarted","Data":"9d5b68c5bb6b53932e3f7da9b2ccf774b80efebd4dc38f852c7b0358d4e280e8"}
Sep 15 16:27:31.271691 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.271671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kwhm5" event={"ID":"34b7f025-99b9-4a22-a786-de64866c0071","Type":"ContainerStarted","Data":"bf325af23490734cd04003a67e052dbb3558c5041d42d689f41f7fe5e8a578ec"}
Sep 15 16:27:31.280173 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.280137    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-27.ec2.internal" podStartSLOduration=21.280126695 podStartE2EDuration="21.280126695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.246341138 +0000 UTC m=+5.632980407" watchObservedRunningTime="2026-09-15 16:27:31.280126695 +0000 UTC m=+22.666765958"
Sep 15 16:27:31.305315 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.305268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-whwt5" podStartSLOduration=13.966005894 podStartE2EDuration="22.305253152s" totalImagesPullingTime="8.339247258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.28121439 +0000 UTC m=+22.667853658" watchObservedRunningTime="2026-09-15 16:27:31.305253152 +0000 UTC m=+22.691892418"
Sep 15 16:27:31.339906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.339850    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9snvm" podStartSLOduration=14.884626993 podStartE2EDuration="22.339830147s" totalImagesPullingTime="7.455203154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.305400884 +0000 UTC m=+22.692040140" watchObservedRunningTime="2026-09-15 16:27:31.339830147 +0000 UTC m=+22.726469417"
Sep 15 16:27:31.366822 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.366776    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xsxqb" podStartSLOduration=4.126017369 podStartE2EDuration="22.366760824s" totalImagesPullingTime="18.240743455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.339671495 +0000 UTC m=+22.726310761" watchObservedRunningTime="2026-09-15 16:27:31.366760824 +0000 UTC m=+22.753400098"
Sep 15 16:27:31.418712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.418632    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g28k2" podStartSLOduration=4.213945571 podStartE2EDuration="22.418613697s" totalImagesPullingTime="18.204668126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.417975263 +0000 UTC m=+22.804614549" watchObservedRunningTime="2026-09-15 16:27:31.418613697 +0000 UTC m=+22.805252954"
Sep 15 16:27:31.482158 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.482110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kwhm5" podStartSLOduration=4.290757283 podStartE2EDuration="22.482095706s" totalImagesPullingTime="18.191338423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.481880256 +0000 UTC m=+22.868519524" watchObservedRunningTime="2026-09-15 16:27:31.482095706 +0000 UTC m=+22.868734973"
Sep 15 16:27:31.823828 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:31.823667    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 16:27:32.092699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:32.092596    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T16:27:31.823821935Z","UUID":"8d9ebc72-1119-4391-bf1e-4793396c8034","Handler":null,"Name":"","Endpoint":""}
Sep 15 16:27:32.094330 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:32.094308    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 16:27:32.094449 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:32.094344    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 16:27:32.275620 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:32.275526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d52kx" event={"ID":"54a962b6-0a66-47b4-8205-4bd659c5c8e0","Type":"ContainerStarted","Data":"392fb9a5937cb4105cec3b986b2d520313841414a7a9e4e9ef62856567b6f442"}
Sep 15 16:27:32.277288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:32.277263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552" event={"ID":"19193dc6-7d7e-4ecb-affa-a9879ec38e90","Type":"ContainerStarted","Data":"070a48d33207a6e6082e891f3a53a077ba96d1f27e03f50e88efb2df192bebe4"}
Sep 15 16:27:33.148725 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.148701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:33.148863 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:33.148839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:33.148933 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.148919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:33.149022 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:33.149003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:33.149105 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.149092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:33.149187 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:33.149170    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:33.284313 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.284231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"93a202f1f6be2925e8cb771dc77fda172c9c6958e50dc0574e98c97f8461af8c"}
Sep 15 16:27:33.287623 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.287591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552" event={"ID":"19193dc6-7d7e-4ecb-affa-a9879ec38e90","Type":"ContainerStarted","Data":"8158fadb21f856f0ada75539b6ccd28d8c4b6207d20e265711f9240cc267e830"}
Sep 15 16:27:33.319731 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.319683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d52kx" podStartSLOduration=6.121828436 podStartE2EDuration="24.319667776s" totalImagesPullingTime="18.19783934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:32.296045381 +0000 UTC m=+23.682684649" watchObservedRunningTime="2026-09-15 16:27:33.319667776 +0000 UTC m=+24.706307043"
Sep 15 16:27:33.320129 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:33.320105    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cz552" podStartSLOduration=11.054684342 podStartE2EDuration="24.320096047s" totalImagesPullingTime="13.265411705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:33.320082628 +0000 UTC m=+24.706721905" watchObservedRunningTime="2026-09-15 16:27:33.320096047 +0000 UTC m=+24.706735314"
Sep 15 16:27:34.783343 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:34.783298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:34.784047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:34.784013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:35.149666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.149636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:35.149666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.149653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:35.149914 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:35.149788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:35.150130 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.149636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:35.150248 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:35.150228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:35.150324 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:35.150304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:35.300894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.300392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" event={"ID":"a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590","Type":"ContainerStarted","Data":"061a9dfb65c570bab41905a6c2376729f43c7765939a45ad4a0d05ea9bd5edd5"}
Sep 15 16:27:35.300894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.300510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:35.302000 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.301633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9snvm"
Sep 15 16:27:35.335774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:35.330484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt" podStartSLOduration=8.078963721 podStartE2EDuration="26.330465851s" totalImagesPullingTime="18.25150213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:35.330219197 +0000 UTC m=+26.716858465" watchObservedRunningTime="2026-09-15 16:27:35.330465851 +0000 UTC m=+26.717105126"
Sep 15 16:27:36.304138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:36.303685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:36.304138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:36.304015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:36.304138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:36.304070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:36.320836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:36.320811    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:36.321049 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:36.321034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:27:37.149522 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.149499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:37.149670 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.149507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:37.149670 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.149600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:37.149670 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.149611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:37.149859 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.149720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:37.149859 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.149720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:37.310558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.310510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerStarted","Data":"6d48c3cc889bbb88b39c63b9870f4c70d1da0b70cef832c75da5f2e9ba6ad837"}
Sep 15 16:27:37.680467 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.680434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjswr"]
Sep 15 16:27:37.680641 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.680575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:37.680691 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.680665    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:37.688854 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.688828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tpn5j"]
Sep 15 16:27:37.688969 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.688925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:37.689013 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.688996    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:37.694805 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.694783    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5g8m"]
Sep 15 16:27:37.694919 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:37.694871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:37.694964 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:37.694946    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:38.314208 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:38.314031    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="6d48c3cc889bbb88b39c63b9870f4c70d1da0b70cef832c75da5f2e9ba6ad837" exitCode=0
Sep 15 16:27:38.314663 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:38.314118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"6d48c3cc889bbb88b39c63b9870f4c70d1da0b70cef832c75da5f2e9ba6ad837"}
Sep 15 16:27:39.048149 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:39.048118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kwhm5_34b7f025-99b9-4a22-a786-de64866c0071/dns-node-resolver/0.log"
Sep 15 16:27:39.150728 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:39.150693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:39.150913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:39.150830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:39.150913 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:39.150846    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:39.150913 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:39.150860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:39.151015 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:39.150933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:39.151046 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:39.151016    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:39.819441 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:39.819415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whwt5_932e1943-4d3d-42d7-bab6-7e4270936ec4/node-ca/0.log"
Sep 15 16:27:40.322978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:40.322938    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="ead860c5aa6aa2814e7e48c4d60db252285c2cacd5c86e18fdc4738d149ddc53" exitCode=0
Sep 15 16:27:40.323134 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:40.322985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"ead860c5aa6aa2814e7e48c4d60db252285c2cacd5c86e18fdc4738d149ddc53"}
Sep 15 16:27:41.149501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:41.149468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:41.149905 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:41.149507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:41.149905 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:41.149467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:41.149905 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:41.149613    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:41.149905 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:41.149631    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:41.149905 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:41.149685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:42.330032 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:42.330002    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="abeedba6c3162377c0d78d8539da03f08edf8ffee424dedded096f444c7abd0f" exitCode=0
Sep 15 16:27:42.330461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:42.330047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"abeedba6c3162377c0d78d8539da03f08edf8ffee424dedded096f444c7abd0f"}
Sep 15 16:27:42.776888 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:42.776849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:42.777063 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.776977    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:42.777063 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.777045    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs podName:b15f4535-2258-4631-9616-f79019a4f309 nodeName:}" failed. No retries permitted until 2026-09-15 16:28:14.777025585 +0000 UTC m=+66.163664846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs") pod "network-metrics-daemon-sjswr" (UID: "b15f4535-2258-4631-9616-f79019a4f309") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:42.877781 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:42.877722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:42.877931 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.877845    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:42.877931 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.877909    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret podName:a16f1e9a-a3bb-450f-881a-15e066a3b639 nodeName:}" failed. No retries permitted until 2026-09-15 16:28:14.877892011 +0000 UTC m=+66.264531257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret") pod "global-pull-secret-syncer-tpn5j" (UID: "a16f1e9a-a3bb-450f-881a-15e066a3b639") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:42.979045 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:42.979005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:42.979221 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.979191    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:42.979221 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.979215    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:42.979361 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.979227    2578 projected.go:196] Error preparing data for projected volume kube-api-access-nsckc for pod openshift-network-diagnostics/network-check-target-q5g8m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:42.979361 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:42.979295    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc podName:656cb863-0d56-4cec-838c-6f8a0bbdba5e nodeName:}" failed. No retries permitted until 2026-09-15 16:28:14.979276456 +0000 UTC m=+66.365915718 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nsckc" (UniqueName: "kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc") pod "network-check-target-q5g8m" (UID: "656cb863-0d56-4cec-838c-6f8a0bbdba5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:43.148799 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:43.148769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:43.148799 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:43.148786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:43.149004 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:43.148769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:43.149004 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:43.148886    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:43.149004 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:43.148954    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:43.149102 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:43.149019    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:45.152267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:45.152242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:45.152267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:45.152256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:45.152718 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:45.152256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:45.152718 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:45.152374    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:45.152718 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:45.152439    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:45.152718 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:45.152512    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:47.152222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:47.152051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:47.152571 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:47.152051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:47.152571 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:47.152310    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:47.152571 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:47.152395    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:47.152571 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:47.152082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:47.152571 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:47.152488    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:47.344474 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:47.344442    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="8d66652b1ba96332895b989cb7402d3b858255717f0b71189f3b3ab5a3d16603" exitCode=0
Sep 15 16:27:47.344622 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:47.344485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"8d66652b1ba96332895b989cb7402d3b858255717f0b71189f3b3ab5a3d16603"}
Sep 15 16:27:48.349573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:48.349536    2578 generic.go:356] "Generic (PLEG): container finished" podID="94305cb9-6f3f-4231-9211-c4bac29600f7" containerID="d3153cffa4860e48ab89484f7fd2c19e23ea363e8d630945ac0a7cc6d37e4d84" exitCode=0
Sep 15 16:27:48.350027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:48.349599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerDied","Data":"d3153cffa4860e48ab89484f7fd2c19e23ea363e8d630945ac0a7cc6d37e4d84"}
Sep 15 16:27:49.150016 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:49.149975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:49.150172 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:49.150076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:49.150172 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:49.150085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:49.150172 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:49.150139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:49.150275 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:49.150258    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:49.150337 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:49.150320    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:49.355047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:49.355011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" event={"ID":"94305cb9-6f3f-4231-9211-c4bac29600f7","Type":"ContainerStarted","Data":"d759a2ec899d1109a33cdb9e0989166d2328648fe2f1ebe41e3e7e0f951689c3"}
Sep 15 16:27:49.378809 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:49.378759    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4nsv5" podStartSLOduration=9.493083044 podStartE2EDuration="40.378725122s" totalImagesPullingTime="30.885642078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:49.378290281 +0000 UTC m=+40.764929572" watchObservedRunningTime="2026-09-15 16:27:49.378725122 +0000 UTC m=+40.765364389"
Sep 15 16:27:51.149043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:51.149004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:51.149463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:51.149004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:51.149463 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:51.149121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:51.149463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:51.149015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:51.149463 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:51.149211    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:51.149463 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:51.149260    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:53.148950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:53.148913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:53.149337 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:53.148925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:53.149337 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:53.149016    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:53.149337 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:53.149082    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:53.149337 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:53.149106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:53.149337 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:53.149203    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:55.149004 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:55.148968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:55.149558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:55.148968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:55.149558 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:55.149086    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:55.149558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:55.149100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:55.149558 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:55.149221    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:55.149558 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:55.149278    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:57.148879 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:57.148846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:57.148879 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:57.148864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:57.149392 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:57.148961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:27:57.149392 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:57.148997    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:57.149392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:57.149031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:57.149392 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:57.149101    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:59.149437 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:59.149405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:27:59.149919 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:59.149499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:27:59.149919 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:59.149531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:27:59.149919 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:59.149605    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:27:59.149919 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:27:59.149531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:27:59.149919 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:27:59.149673    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:28:01.149032 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:01.148993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:01.149486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:01.149001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:01.149486 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:01.149112    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:28:01.149486 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:01.149163    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:28:01.149486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:01.149184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:01.149486 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:01.149303    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:28:03.149224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:03.149194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:03.149224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:03.149216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:03.149699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:03.149199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:03.149699 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:03.149290    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:28:03.149699 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:03.149368    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:28:03.149699 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:03.149473    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:28:05.149231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:05.149200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:05.149573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:05.149200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:05.149573 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:05.149323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjswr" podUID="b15f4535-2258-4631-9616-f79019a4f309"
Sep 15 16:28:05.149573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:05.149426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:05.149573 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:05.149433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tpn5j" podUID="a16f1e9a-a3bb-450f-881a-15e066a3b639"
Sep 15 16:28:05.149573 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:05.149485    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5g8m" podUID="656cb863-0d56-4cec-838c-6f8a0bbdba5e"
Sep 15 16:28:06.383217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.383143    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-27.ec2.internal" event="NodeReady"
Sep 15 16:28:06.383624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.383254    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 16:28:06.496831 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.496796    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pz7zt"]
Sep 15 16:28:06.498717 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.498699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.505066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.504938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzxqz\""
Sep 15 16:28:06.505254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.505235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 16:28:06.510002 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.509987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 16:28:06.514723 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.514701    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8s5gr"]
Sep 15 16:28:06.516314 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.516297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.521046 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.521026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz7zt"]
Sep 15 16:28:06.525794 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.525770    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 16:28:06.525947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.525931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn9ff\""
Sep 15 16:28:06.526011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.525986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 16:28:06.529664 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.529645    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 16:28:06.543903 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.543875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8s5gr"]
Sep 15 16:28:06.588542 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.588511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l6547"]
Sep 15 16:28:06.590495 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.590480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.596846 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.596824    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 16:28:06.596846 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.596839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 16:28:06.596999 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.596839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 16:28:06.596999 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.596831    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wmprm\""
Sep 15 16:28:06.597136 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.597119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 16:28:06.621412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.621381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l6547"]
Sep 15 16:28:06.638965 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.638941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzmc\" (UniqueName: \"kubernetes.io/projected/1332b3c3-c602-48ba-9b74-42833d520538-kube-api-access-wrzmc\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.639054 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.638996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1332b3c3-c602-48ba-9b74-42833d520538-tmp-dir\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.639054 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.639038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfnt\" (UniqueName: \"kubernetes.io/projected/c72c1799-15bb-4c48-944b-ceee49f6af1c-kube-api-access-mmfnt\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.639121 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.639081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1332b3c3-c602-48ba-9b74-42833d520538-config-volume\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.639157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.639118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c72c1799-15bb-4c48-944b-ceee49f6af1c-cert\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.639157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.639135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1332b3c3-c602-48ba-9b74-42833d520538-metrics-tls\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.739607 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzmc\" (UniqueName: \"kubernetes.io/projected/1332b3c3-c602-48ba-9b74-42833d520538-kube-api-access-wrzmc\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.739776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/653f724c-fdde-40a4-8b73-876d52c2cc26-crio-socket\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.739776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1332b3c3-c602-48ba-9b74-42833d520538-tmp-dir\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.739776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfnt\" (UniqueName: \"kubernetes.io/projected/c72c1799-15bb-4c48-944b-ceee49f6af1c-kube-api-access-mmfnt\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.739776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.739776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1332b3c3-c602-48ba-9b74-42833d520538-config-volume\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.739979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/653f724c-fdde-40a4-8b73-876d52c2cc26-data-volume\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.739979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c72c1799-15bb-4c48-944b-ceee49f6af1c-cert\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.739979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1332b3c3-c602-48ba-9b74-42833d520538-metrics-tls\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.740126 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.739996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/653f724c-fdde-40a4-8b73-876d52c2cc26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.740126 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.740034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44hq\" (UniqueName: \"kubernetes.io/projected/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-api-access-x44hq\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.740126 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.740087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1332b3c3-c602-48ba-9b74-42833d520538-tmp-dir\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.740305 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.740289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1332b3c3-c602-48ba-9b74-42833d520538-config-volume\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.743840 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.743817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c72c1799-15bb-4c48-944b-ceee49f6af1c-cert\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.743840 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.743824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1332b3c3-c602-48ba-9b74-42833d520538-metrics-tls\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.751128 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.751108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfnt\" (UniqueName: \"kubernetes.io/projected/c72c1799-15bb-4c48-944b-ceee49f6af1c-kube-api-access-mmfnt\") pod \"ingress-canary-8s5gr\" (UID: \"c72c1799-15bb-4c48-944b-ceee49f6af1c\") " pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.751242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.751225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzmc\" (UniqueName: \"kubernetes.io/projected/1332b3c3-c602-48ba-9b74-42833d520538-kube-api-access-wrzmc\") pod \"dns-default-pz7zt\" (UID: \"1332b3c3-c602-48ba-9b74-42833d520538\") " pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.807226 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.807193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:06.824126 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.824104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8s5gr"
Sep 15 16:28:06.840342 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/653f724c-fdde-40a4-8b73-876d52c2cc26-crio-socket\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/653f724c-fdde-40a4-8b73-876d52c2cc26-data-volume\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/653f724c-fdde-40a4-8b73-876d52c2cc26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44hq\" (UniqueName: \"kubernetes.io/projected/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-api-access-x44hq\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/653f724c-fdde-40a4-8b73-876d52c2cc26-crio-socket\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.840835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/653f724c-fdde-40a4-8b73-876d52c2cc26-data-volume\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.841011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.840996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.842925 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.842906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/653f724c-fdde-40a4-8b73-876d52c2cc26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.849910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.849886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44hq\" (UniqueName: \"kubernetes.io/projected/653f724c-fdde-40a4-8b73-876d52c2cc26-kube-api-access-x44hq\") pod \"insights-runtime-extractor-l6547\" (UID: \"653f724c-fdde-40a4-8b73-876d52c2cc26\") " pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:06.899145 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:06.899021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l6547"
Sep 15 16:28:07.087412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.087384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz7zt"]
Sep 15 16:28:07.091185 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:07.091146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1332b3c3_c602_48ba_9b74_42833d520538.slice/crio-4da7c1116e6fa9c2badb33edbed360dc865bf246ce4991332fb90664c6b76c5a WatchSource:0}: Error finding container 4da7c1116e6fa9c2badb33edbed360dc865bf246ce4991332fb90664c6b76c5a: Status 404 returned error can't find the container with id 4da7c1116e6fa9c2badb33edbed360dc865bf246ce4991332fb90664c6b76c5a
Sep 15 16:28:07.103358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.103331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8s5gr"]
Sep 15 16:28:07.106056 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:07.106031    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72c1799_15bb_4c48_944b_ceee49f6af1c.slice/crio-cd28b2b8069be2aabaf19ace857a355d76525e89c9837476a87b6dd8f58932a6 WatchSource:0}: Error finding container cd28b2b8069be2aabaf19ace857a355d76525e89c9837476a87b6dd8f58932a6: Status 404 returned error can't find the container with id cd28b2b8069be2aabaf19ace857a355d76525e89c9837476a87b6dd8f58932a6
Sep 15 16:28:07.148912 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.148888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:07.148912 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.148901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:07.149082 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.148896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:07.153722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.153247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:28:07.153722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.153412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:28:07.153722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.153491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:28:07.154224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.154187    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhmxd\""
Sep 15 16:28:07.154224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.154198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:28:07.154393 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.154203    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkdt8\""
Sep 15 16:28:07.170308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.170283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l6547"]
Sep 15 16:28:07.173648 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:07.173624    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653f724c_fdde_40a4_8b73_876d52c2cc26.slice/crio-0a555972896ad3c7cef6aaed5ecfa98f2efdc2d56b19235d3409369cb0ca6424 WatchSource:0}: Error finding container 0a555972896ad3c7cef6aaed5ecfa98f2efdc2d56b19235d3409369cb0ca6424: Status 404 returned error can't find the container with id 0a555972896ad3c7cef6aaed5ecfa98f2efdc2d56b19235d3409369cb0ca6424
Sep 15 16:28:07.394960 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.394918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6547" event={"ID":"653f724c-fdde-40a4-8b73-876d52c2cc26","Type":"ContainerStarted","Data":"95ecf293b82a5425b736f89c95127ac6381c6ed991e96f6b43d9ad2c88728039"}
Sep 15 16:28:07.395358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.394978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6547" event={"ID":"653f724c-fdde-40a4-8b73-876d52c2cc26","Type":"ContainerStarted","Data":"0a555972896ad3c7cef6aaed5ecfa98f2efdc2d56b19235d3409369cb0ca6424"}
Sep 15 16:28:07.396147 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.396125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8s5gr" event={"ID":"c72c1799-15bb-4c48-944b-ceee49f6af1c","Type":"ContainerStarted","Data":"cd28b2b8069be2aabaf19ace857a355d76525e89c9837476a87b6dd8f58932a6"}
Sep 15 16:28:07.397233 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:07.397212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz7zt" event={"ID":"1332b3c3-c602-48ba-9b74-42833d520538","Type":"ContainerStarted","Data":"4da7c1116e6fa9c2badb33edbed360dc865bf246ce4991332fb90664c6b76c5a"}
Sep 15 16:28:08.329400 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:08.329320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtgkt"
Sep 15 16:28:08.402337 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:08.402267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6547" event={"ID":"653f724c-fdde-40a4-8b73-876d52c2cc26","Type":"ContainerStarted","Data":"1990fcf96f7703a9402ef615d2d8e5317f7ff36d7ef260b3b1b920ea3cbcd2eb"}
Sep 15 16:28:09.407773 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:09.407488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz7zt" event={"ID":"1332b3c3-c602-48ba-9b74-42833d520538","Type":"ContainerStarted","Data":"88103386cb924d80f10f1af6ca2870dd8cd8d22b01a58baabfa286d0d705c0c9"}
Sep 15 16:28:09.407773 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:09.407555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz7zt" event={"ID":"1332b3c3-c602-48ba-9b74-42833d520538","Type":"ContainerStarted","Data":"669691aed5c80f3cfdbefe4ec8d6d73305b25f1fb3551f03d5cc1001c70ea603"}
Sep 15 16:28:09.407773 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:09.407596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:09.425996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:09.425518    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pz7zt" podStartSLOduration=1.582048932 podStartE2EDuration="3.425503543s" totalImagesPullingTime="1.843454611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.424561108 +0000 UTC m=+60.811200376" watchObservedRunningTime="2026-09-15 16:28:09.425503543 +0000 UTC m=+60.812142804"
Sep 15 16:28:11.092656 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.092616    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"]
Sep 15 16:28:11.094712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.094694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.097254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.097229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 16:28:11.097254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.097251    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 16:28:11.098191 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.098171    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 16:28:11.098299 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.098194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 16:28:11.098299 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.098220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bxbbb\""
Sep 15 16:28:11.098299 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.098175    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 16:28:11.104779 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.104725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"]
Sep 15 16:28:11.225860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.225829    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-x5vt9"]
Sep 15 16:28:11.227747 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.227722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:11.230140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.230118    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 16:28:11.230262 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.230247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 16:28:11.230262 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.230255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m2tx8\""
Sep 15 16:28:11.242697 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.242677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-x5vt9"]
Sep 15 16:28:11.273212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.273182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mjx\" (UniqueName: \"kubernetes.io/projected/a84d5b9b-d058-4181-9a78-4218703965dc-kube-api-access-h2mjx\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.273324 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.273232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.273324 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.273257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84d5b9b-d058-4181-9a78-4218703965dc-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.273401 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.273313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.374163 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mjx\" (UniqueName: \"kubernetes.io/projected/a84d5b9b-d058-4181-9a78-4218703965dc-kube-api-access-h2mjx\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.374163 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.374163 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84d5b9b-d058-4181-9a78-4218703965dc-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.374390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.374390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6ff\" (UniqueName: \"kubernetes.io/projected/9ca42511-83bd-42a7-a913-d80732f74f6e-kube-api-access-wx6ff\") pod \"downloads-76f99584f4-x5vt9\" (UID: \"9ca42511-83bd-42a7-a913-d80732f74f6e\") " pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:11.374863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.374834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84d5b9b-d058-4181-9a78-4218703965dc-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.376784 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.376760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.377374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.377356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a84d5b9b-d058-4181-9a78-4218703965dc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.382693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.382674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mjx\" (UniqueName: \"kubernetes.io/projected/a84d5b9b-d058-4181-9a78-4218703965dc-kube-api-access-h2mjx\") pod \"prometheus-operator-68bcd7f987-jqzh7\" (UID: \"a84d5b9b-d058-4181-9a78-4218703965dc\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.403333 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.403311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"
Sep 15 16:28:11.416267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.416232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6547" event={"ID":"653f724c-fdde-40a4-8b73-876d52c2cc26","Type":"ContainerStarted","Data":"8ceb55dddd92942b0039ade4cd30de27de1cd50170942bda75b3c65621a0fb27"}
Sep 15 16:28:11.417826 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.417804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8s5gr" event={"ID":"c72c1799-15bb-4c48-944b-ceee49f6af1c","Type":"ContainerStarted","Data":"c79f91ebafe62cd3878241d875e78bb0f0765ef79acff803dded1be45c5b4775"}
Sep 15 16:28:11.436790 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.436709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l6547" podStartSLOduration=2.282904035 podStartE2EDuration="5.436693337s" totalImagesPullingTime="3.153789302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:28:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:11.435542397 +0000 UTC m=+62.822181665" watchObservedRunningTime="2026-09-15 16:28:11.436693337 +0000 UTC m=+62.823332605"
Sep 15 16:28:11.457332 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.457276    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8s5gr" podStartSLOduration=2.101919532 podStartE2EDuration="5.457258994s" totalImagesPullingTime="3.355339462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:11.456670723 +0000 UTC m=+62.843309992" watchObservedRunningTime="2026-09-15 16:28:11.457258994 +0000 UTC m=+62.843898281"
Sep 15 16:28:11.475614 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.475257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6ff\" (UniqueName: \"kubernetes.io/projected/9ca42511-83bd-42a7-a913-d80732f74f6e-kube-api-access-wx6ff\") pod \"downloads-76f99584f4-x5vt9\" (UID: \"9ca42511-83bd-42a7-a913-d80732f74f6e\") " pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:11.489514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.489477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6ff\" (UniqueName: \"kubernetes.io/projected/9ca42511-83bd-42a7-a913-d80732f74f6e-kube-api-access-wx6ff\") pod \"downloads-76f99584f4-x5vt9\" (UID: \"9ca42511-83bd-42a7-a913-d80732f74f6e\") " pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:11.536037 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.535993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:11.676014 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.675984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7"]
Sep 15 16:28:11.679023 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:11.678993    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84d5b9b_d058_4181_9a78_4218703965dc.slice/crio-a7202ef36ec3ea599b8a1a38a8b42a77924f2541188d3976e4e2bd191119b261 WatchSource:0}: Error finding container a7202ef36ec3ea599b8a1a38a8b42a77924f2541188d3976e4e2bd191119b261: Status 404 returned error can't find the container with id a7202ef36ec3ea599b8a1a38a8b42a77924f2541188d3976e4e2bd191119b261
Sep 15 16:28:11.806984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:11.806906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-x5vt9"]
Sep 15 16:28:11.809305 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:11.809278    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca42511_83bd_42a7_a913_d80732f74f6e.slice/crio-3bd2c0a65e5c604e55f5ad04bf287a6d988950d823979ebf5009a177a20fb89f WatchSource:0}: Error finding container 3bd2c0a65e5c604e55f5ad04bf287a6d988950d823979ebf5009a177a20fb89f: Status 404 returned error can't find the container with id 3bd2c0a65e5c604e55f5ad04bf287a6d988950d823979ebf5009a177a20fb89f
Sep 15 16:28:12.425558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:12.425504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-x5vt9" event={"ID":"9ca42511-83bd-42a7-a913-d80732f74f6e","Type":"ContainerStarted","Data":"3bd2c0a65e5c604e55f5ad04bf287a6d988950d823979ebf5009a177a20fb89f"}
Sep 15 16:28:12.426942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:12.426909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7" event={"ID":"a84d5b9b-d058-4181-9a78-4218703965dc","Type":"ContainerStarted","Data":"a7202ef36ec3ea599b8a1a38a8b42a77924f2541188d3976e4e2bd191119b261"}
Sep 15 16:28:13.433340 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:13.433293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7" event={"ID":"a84d5b9b-d058-4181-9a78-4218703965dc","Type":"ContainerStarted","Data":"26c51a484c2d89ffdfd253cf17e1b687d656ed0057f26f885c8d5baf15c17232"}
Sep 15 16:28:13.433786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:13.433375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7" event={"ID":"a84d5b9b-d058-4181-9a78-4218703965dc","Type":"ContainerStarted","Data":"682714a7ebacb0aaeaeb96bbf786b3a8c1f6dd95e0cdec13ae0392a8f7ae1f46"}
Sep 15 16:28:13.452467 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:13.452414    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-jqzh7" podStartSLOduration=1.3287941220000001 podStartE2EDuration="2.452399915s" totalImagesPullingTime="1.123605793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:13.451761167 +0000 UTC m=+64.838400432" watchObservedRunningTime="2026-09-15 16:28:13.452399915 +0000 UTC m=+64.839039181"
Sep 15 16:28:14.802157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.802115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:14.804910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.804884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:28:14.815558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.815531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15f4535-2258-4631-9616-f79019a4f309-metrics-certs\") pod \"network-metrics-daemon-sjswr\" (UID: \"b15f4535-2258-4631-9616-f79019a4f309\") " pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:14.903141 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.903103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:14.907420 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.907402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:28:14.916443 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.916422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a16f1e9a-a3bb-450f-881a-15e066a3b639-original-pull-secret\") pod \"global-pull-secret-syncer-tpn5j\" (UID: \"a16f1e9a-a3bb-450f-881a-15e066a3b639\") " pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:14.969291 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.969260    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkdt8\""
Sep 15 16:28:14.971251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.971221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tpn5j"
Sep 15 16:28:14.977797 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:14.977779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjswr"
Sep 15 16:28:15.003833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.003726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:15.006403 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.006323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:28:15.017375 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.017352    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:28:15.029558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.029378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsckc\" (UniqueName: \"kubernetes.io/projected/656cb863-0d56-4cec-838c-6f8a0bbdba5e-kube-api-access-nsckc\") pod \"network-check-target-q5g8m\" (UID: \"656cb863-0d56-4cec-838c-6f8a0bbdba5e\") " pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:15.261586 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.261558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tpn5j"]
Sep 15 16:28:15.262981 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.262957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhmxd\""
Sep 15 16:28:15.264768 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:15.264207    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16f1e9a_a3bb_450f_881a_15e066a3b639.slice/crio-f4de82241c69c3a847784d8eeaa2a94d8c63fc24a60405af2ddc3ff1cff93172 WatchSource:0}: Error finding container f4de82241c69c3a847784d8eeaa2a94d8c63fc24a60405af2ddc3ff1cff93172: Status 404 returned error can't find the container with id f4de82241c69c3a847784d8eeaa2a94d8c63fc24a60405af2ddc3ff1cff93172
Sep 15 16:28:15.264768 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.264426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjswr"]
Sep 15 16:28:15.268555 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:15.268536    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15f4535_2258_4631_9616_f79019a4f309.slice/crio-421ba7551aac564bc44ddd0f45717e5a1fa609adb0dce773c58b30c40366bc91 WatchSource:0}: Error finding container 421ba7551aac564bc44ddd0f45717e5a1fa609adb0dce773c58b30c40366bc91: Status 404 returned error can't find the container with id 421ba7551aac564bc44ddd0f45717e5a1fa609adb0dce773c58b30c40366bc91
Sep 15 16:28:15.270684 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.270665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:15.440812 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.440771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjswr" event={"ID":"b15f4535-2258-4631-9616-f79019a4f309","Type":"ContainerStarted","Data":"421ba7551aac564bc44ddd0f45717e5a1fa609adb0dce773c58b30c40366bc91"}
Sep 15 16:28:15.442265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.442237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tpn5j" event={"ID":"a16f1e9a-a3bb-450f-881a-15e066a3b639","Type":"ContainerStarted","Data":"f4de82241c69c3a847784d8eeaa2a94d8c63fc24a60405af2ddc3ff1cff93172"}
Sep 15 16:28:15.459361 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.459331    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"]
Sep 15 16:28:15.462123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.462103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.465376 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.465351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 16:28:15.465484 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.465454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g8nw5\""
Sep 15 16:28:15.465624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.465594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 16:28:15.477790 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.477770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"]
Sep 15 16:28:15.496358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.496324    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"]
Sep 15 16:28:15.498810 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.498789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.501764 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.501723    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 16:28:15.503295 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.502524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wdfz6\""
Sep 15 16:28:15.503295 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.502633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 16:28:15.503295 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.503151    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 16:28:15.506971 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.506948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkg4\" (UniqueName: \"kubernetes.io/projected/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-kube-api-access-cjkg4\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.507084 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.506991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.507084 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.507084 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.507242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.507242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.507242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8tz\" (UniqueName: \"kubernetes.io/projected/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-api-access-lh8tz\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.507242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.507242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c30c7ef-0a71-417a-b265-4265ac14fa19-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.507418 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.507251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.518189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.518038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"]
Sep 15 16:28:15.545212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.545178    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6xs2s"]
Sep 15 16:28:15.548174 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.548149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.551579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.551555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 16:28:15.551678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.551589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 16:28:15.551936 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.551909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 16:28:15.552213 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.552197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ttpl\""
Sep 15 16:28:15.573764 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.572809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5g8m"]
Sep 15 16:28:15.575765 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:15.575715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656cb863_0d56_4cec_838c_6f8a0bbdba5e.slice/crio-172ca6e90084c52df20a62efc5f40bbfd628ff1d2896dde74b55c1f237ee23f7 WatchSource:0}: Error finding container 172ca6e90084c52df20a62efc5f40bbfd628ff1d2896dde74b55c1f237ee23f7: Status 404 returned error can't find the container with id 172ca6e90084c52df20a62efc5f40bbfd628ff1d2896dde74b55c1f237ee23f7
Sep 15 16:28:15.608347 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-sys\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:15.608505    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-textfile\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:15.608564    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls podName:7c30c7ef-0a71-417a-b265-4265ac14fa19 nodeName:}" failed. No retries permitted until 2026-09-15 16:28:16.108544731 +0000 UTC m=+67.495183977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-l5sw2" (UID: "7c30c7ef-0a71-417a-b265-4265ac14fa19") : secret "kube-state-metrics-tls" not found
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c30c7ef-0a71-417a-b265-4265ac14fa19-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-root\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzcv\" (UniqueName: \"kubernetes.io/projected/ebd7b653-078c-45b6-9676-100d8f295184-kube-api-access-nkzcv\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-tls\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-wtmp\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8tz\" (UniqueName: \"kubernetes.io/projected/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-api-access-lh8tz\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-metrics-client-ca\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.608984 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.608911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkg4\" (UniqueName: \"kubernetes.io/projected/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-kube-api-access-cjkg4\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.609374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.609092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7c30c7ef-0a71-417a-b265-4265ac14fa19-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.609374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.609152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.609374 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:15.609212    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 16:28:15.609374 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:15.609247    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls podName:5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a nodeName:}" failed. No retries permitted until 2026-09-15 16:28:16.109235766 +0000 UTC m=+67.495875011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-psnts" (UID: "5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a") : secret "openshift-state-metrics-tls" not found
Sep 15 16:28:15.609510 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.609374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.609841 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.609817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.611769 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.611721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.611863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.611792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.621839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.621819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkg4\" (UniqueName: \"kubernetes.io/projected/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-kube-api-access-cjkg4\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:15.622550 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.622528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8tz\" (UniqueName: \"kubernetes.io/projected/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-api-access-lh8tz\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:15.710220 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-wtmp\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710220 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-metrics-client-ca\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-wtmp\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710572 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-sys\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710572 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-sys\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710572 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-textfile\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-root\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzcv\" (UniqueName: \"kubernetes.io/projected/ebd7b653-078c-45b6-9676-100d8f295184-kube-api-access-nkzcv\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-tls\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710886 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-textfile\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebd7b653-078c-45b6-9676-100d8f295184-root\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.710998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.710983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-metrics-client-ca\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.711065 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.711040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.713288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.713261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-tls\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.713426 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.713393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd7b653-078c-45b6-9676-100d8f295184-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.720405 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.720382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzcv\" (UniqueName: \"kubernetes.io/projected/ebd7b653-078c-45b6-9676-100d8f295184-kube-api-access-nkzcv\") pod \"node-exporter-6xs2s\" (UID: \"ebd7b653-078c-45b6-9676-100d8f295184\") " pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.858952 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:15.858870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xs2s"
Sep 15 16:28:15.874719 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:15.874686    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd7b653_078c_45b6_9676_100d8f295184.slice/crio-bf37c56b473f69f802c4a35d5369c98581d55994c92c17044893c5f1dca148c7 WatchSource:0}: Error finding container bf37c56b473f69f802c4a35d5369c98581d55994c92c17044893c5f1dca148c7: Status 404 returned error can't find the container with id bf37c56b473f69f802c4a35d5369c98581d55994c92c17044893c5f1dca148c7
Sep 15 16:28:16.115687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.115222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:16.115687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.115301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:16.120227 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.120175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c30c7ef-0a71-417a-b265-4265ac14fa19-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-l5sw2\" (UID: \"7c30c7ef-0a71-417a-b265-4265ac14fa19\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:16.120790 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.120730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-psnts\" (UID: \"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:16.372491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.372407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"
Sep 15 16:28:16.412883 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.412851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"
Sep 15 16:28:16.450980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.450894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5g8m" event={"ID":"656cb863-0d56-4cec-838c-6f8a0bbdba5e","Type":"ContainerStarted","Data":"172ca6e90084c52df20a62efc5f40bbfd628ff1d2896dde74b55c1f237ee23f7"}
Sep 15 16:28:16.455943 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.455860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xs2s" event={"ID":"ebd7b653-078c-45b6-9676-100d8f295184","Type":"ContainerStarted","Data":"bf37c56b473f69f802c4a35d5369c98581d55994c92c17044893c5f1dca148c7"}
Sep 15 16:28:16.567263 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.567230    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:28:16.582035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.581952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.584907 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.584881    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 16:28:16.585844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.585311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bjrpj\""
Sep 15 16:28:16.585844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.585549    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 16:28:16.585844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.585766    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 16:28:16.588575 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.586126    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 16:28:16.593821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.590845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 16:28:16.593919 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.593884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 16:28:16.594163 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.594084    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 16:28:16.594241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.594168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 16:28:16.595793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.594088    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 16:28:16.597204 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.597109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sthp\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.619980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.620181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.620028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.693712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.693639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts"]
Sep 15 16:28:16.696400 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:16.696368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff148b5_87bc_4a6a_b4a2_8b50c9511d5a.slice/crio-40b392641f9d344299cf38a89e17e8a7fcb08f35c2b2b467d1c81e1695b552fd WatchSource:0}: Error finding container 40b392641f9d344299cf38a89e17e8a7fcb08f35c2b2b467d1c81e1695b552fd: Status 404 returned error can't find the container with id 40b392641f9d344299cf38a89e17e8a7fcb08f35c2b2b467d1c81e1695b552fd
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sthp\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.720934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.722247 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.721778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:16.723202    2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:16.723265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls podName:074af8b5-cbdf-41b5-bf28-a39d63440c15 nodeName:}" failed. No retries permitted until 2026-09-15 16:28:17.223245959 +0000 UTC m=+68.609885218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15") : secret "alertmanager-main-tls" not found
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:28:16.723480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle podName:074af8b5-cbdf-41b5-bf28-a39d63440c15 nodeName:}" failed. No retries permitted until 2026-09-15 16:28:17.223459642 +0000 UTC m=+68.610098890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15") : configmap references non-existent config key: ca-bundle.crt
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.724972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.729017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.729155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.729109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.729548 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.729344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.732194 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.731415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.732779 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.732483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2"]
Sep 15 16:28:16.734058 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.733475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.734058 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.733491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.734058 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.733862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.734058 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.734032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.735944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:16.735699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sthp\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:16.738005 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:16.737962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c30c7ef_0a71_417a_b265_4265ac14fa19.slice/crio-deabd4f3f4a050f12106f6c3b46ae673a4789e9ae3fd14ae7fb3e6f2aaea7f53 WatchSource:0}: Error finding container deabd4f3f4a050f12106f6c3b46ae673a4789e9ae3fd14ae7fb3e6f2aaea7f53: Status 404 returned error can't find the container with id deabd4f3f4a050f12106f6c3b46ae673a4789e9ae3fd14ae7fb3e6f2aaea7f53
Sep 15 16:28:17.226294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.226246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:17.226770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.226318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:17.227595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.227513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:17.229311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.229285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:17.462427 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.462384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2" event={"ID":"7c30c7ef-0a71-417a-b265-4265ac14fa19","Type":"ContainerStarted","Data":"deabd4f3f4a050f12106f6c3b46ae673a4789e9ae3fd14ae7fb3e6f2aaea7f53"}
Sep 15 16:28:17.464970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.464941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts" event={"ID":"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a","Type":"ContainerStarted","Data":"108d5dd39c416533849f06f5a5fb2c10211837ef566b17a9b7d3f51cc5f8114a"}
Sep 15 16:28:17.465116 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.464992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts" event={"ID":"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a","Type":"ContainerStarted","Data":"40b392641f9d344299cf38a89e17e8a7fcb08f35c2b2b467d1c81e1695b552fd"}
Sep 15 16:28:17.502157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:17.502061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:28:18.172513 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.172478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:28:18.178920 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:18.178875    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074af8b5_cbdf_41b5_bf28_a39d63440c15.slice/crio-b1b763af521d95e28a310f9a3a93a5a9e364771dcdd162db9a68296d0e6f36d0 WatchSource:0}: Error finding container b1b763af521d95e28a310f9a3a93a5a9e364771dcdd162db9a68296d0e6f36d0: Status 404 returned error can't find the container with id b1b763af521d95e28a310f9a3a93a5a9e364771dcdd162db9a68296d0e6f36d0
Sep 15 16:28:18.471477 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.469619    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67559bd77c-77n4v"]
Sep 15 16:28:18.474369 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.474297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjswr" event={"ID":"b15f4535-2258-4631-9616-f79019a4f309","Type":"ContainerStarted","Data":"02c6671426c680c7dea769f21a607126aec1eab4ba8e3f280b61f884b1e128f2"}
Sep 15 16:28:18.474369 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.474339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjswr" event={"ID":"b15f4535-2258-4631-9616-f79019a4f309","Type":"ContainerStarted","Data":"2fa7be5ca4b2646fcb6e523db698fb5c5ecdf49e936ac6f58609229792f184e2"}
Sep 15 16:28:18.474618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.474438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.479928 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.479889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts" event={"ID":"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a","Type":"ContainerStarted","Data":"2100982f3215c709d95130da30f6b71135feb5e4803fcce7bfabb67b884c58b4"}
Sep 15 16:28:18.481491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.481462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"b1b763af521d95e28a310f9a3a93a5a9e364771dcdd162db9a68296d0e6f36d0"}
Sep 15 16:28:18.484114 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.484012    2578 generic.go:356] "Generic (PLEG): container finished" podID="ebd7b653-078c-45b6-9676-100d8f295184" containerID="3d1a0cca964c6ac3fab3cdd54fa9eb6fbb0708c07bca93c1d719413b5b10e9b4" exitCode=0
Sep 15 16:28:18.484114 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.484065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xs2s" event={"ID":"ebd7b653-078c-45b6-9676-100d8f295184","Type":"ContainerDied","Data":"3d1a0cca964c6ac3fab3cdd54fa9eb6fbb0708c07bca93c1d719413b5b10e9b4"}
Sep 15 16:28:18.486513 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.486491    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 16:28:18.486840 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.486809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 16:28:18.487554 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.487089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 16:28:18.487554 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.487339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 16:28:18.488204 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.488183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 16:28:18.488942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.488922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-njkjw\""
Sep 15 16:28:18.497512 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.497484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6vkrrrumg85af\""
Sep 15 16:28:18.508311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.508288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67559bd77c-77n4v"]
Sep 15 16:28:18.544106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.544371 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.544453 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8h4\" (UniqueName: \"kubernetes.io/projected/68b7a40b-3afc-4e5d-aa80-53d17c07500c-kube-api-access-hr8h4\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.544453 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.544551 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.546092 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b7a40b-3afc-4e5d-aa80-53d17c07500c-metrics-client-ca\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.546092 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.546092 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.544697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-grpc-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.552814 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.552438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sjswr" podStartSLOduration=66.964442656 podStartE2EDuration="1m9.552416133s" totalImagesPullingTime="2.587973477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:18.524777284 +0000 UTC m=+69.911416552" watchObservedRunningTime="2026-09-15 16:28:18.552416133 +0000 UTC m=+69.939055402"
Sep 15 16:28:18.645162 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-grpc-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8h4\" (UniqueName: \"kubernetes.io/projected/68b7a40b-3afc-4e5d-aa80-53d17c07500c-kube-api-access-hr8h4\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.645599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.645407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b7a40b-3afc-4e5d-aa80-53d17c07500c-metrics-client-ca\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.646195 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.646169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68b7a40b-3afc-4e5d-aa80-53d17c07500c-metrics-client-ca\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.652627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.652573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.653808 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.653784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-grpc-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.654357 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.654332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-tls\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.654856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.654810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.655494 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.655467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.657051 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.656951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/68b7a40b-3afc-4e5d-aa80-53d17c07500c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.662942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.662917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8h4\" (UniqueName: \"kubernetes.io/projected/68b7a40b-3afc-4e5d-aa80-53d17c07500c-kube-api-access-hr8h4\") pod \"thanos-querier-67559bd77c-77n4v\" (UID: \"68b7a40b-3afc-4e5d-aa80-53d17c07500c\") " pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:18.788146 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:18.787797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:19.413457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.413423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pz7zt"
Sep 15 16:28:19.495851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.495812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xs2s" event={"ID":"ebd7b653-078c-45b6-9676-100d8f295184","Type":"ContainerStarted","Data":"d73af3877810d3df37c4c2a836b168f918ce27798fed351ed39d07b319c174a1"}
Sep 15 16:28:19.496280 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.495878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xs2s" event={"ID":"ebd7b653-078c-45b6-9676-100d8f295184","Type":"ContainerStarted","Data":"239c533d82b387be6436833355949071b249605caf2d8c3d0ab461052c578b21"}
Sep 15 16:28:19.890192 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.890136    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6xs2s" podStartSLOduration=2.781104564 podStartE2EDuration="4.890111483s" totalImagesPullingTime="1.990962339s" totalInitContainerRuntime="118.04458ms" isStatefulPod=true podCreationTimestamp="2026-09-15 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:19.515976737 +0000 UTC m=+70.902616006" watchObservedRunningTime="2026-09-15 16:28:19.890111483 +0000 UTC m=+71.276750751"
Sep 15 16:28:19.890682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.890656    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c7d57d89c-b947v"]
Sep 15 16:28:19.894694 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.894670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.897877 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.897518    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ok3v5cltpgai\""
Sep 15 16:28:19.897996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.897899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kdxf6\""
Sep 15 16:28:19.897996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.897923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 16:28:19.898330 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.898173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 16:28:19.898759 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.898582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 16:28:19.898759 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.898613    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 16:28:19.903293 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.903272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c7d57d89c-b947v"]
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d0ee721d-6d7f-4143-acb1-a4c7e239b884-audit-log\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-client-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-metrics-server-audit-profiles\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-client-certs\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxrj\" (UniqueName: \"kubernetes.io/projected/d0ee721d-6d7f-4143-acb1-a4c7e239b884-kube-api-access-6nxrj\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:19.956636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:19.956519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-tls\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057373 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxrj\" (UniqueName: \"kubernetes.io/projected/d0ee721d-6d7f-4143-acb1-a4c7e239b884-kube-api-access-6nxrj\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057520 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-tls\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057520 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057520 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d0ee721d-6d7f-4143-acb1-a4c7e239b884-audit-log\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057520 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-client-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057665 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-metrics-server-audit-profiles\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.057665 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.057568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-client-certs\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.058441 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.058411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d0ee721d-6d7f-4143-acb1-a4c7e239b884-audit-log\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.058707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.058685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.059424 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.059397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d0ee721d-6d7f-4143-acb1-a4c7e239b884-metrics-server-audit-profiles\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.060881 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.060840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-client-certs\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.060983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.060947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-client-ca-bundle\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.061908 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.061867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d0ee721d-6d7f-4143-acb1-a4c7e239b884-secret-metrics-server-tls\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.066710 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.066691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxrj\" (UniqueName: \"kubernetes.io/projected/d0ee721d-6d7f-4143-acb1-a4c7e239b884-kube-api-access-6nxrj\") pod \"metrics-server-c7d57d89c-b947v\" (UID: \"d0ee721d-6d7f-4143-acb1-a4c7e239b884\") " pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.207222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.207135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:20.218212 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.218188    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"]
Sep 15 16:28:20.220783 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.220761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.223517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.223343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 16:28:20.223747 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.223717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2259d\""
Sep 15 16:28:20.228504 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.228345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"]
Sep 15 16:28:20.258706 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.258664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10f31202-5e4e-4d41-8454-edb20c610fb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.258875 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.258715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwls\" (UniqueName: \"kubernetes.io/projected/10f31202-5e4e-4d41-8454-edb20c610fb8-kube-api-access-4qwls\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.359897 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.359857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10f31202-5e4e-4d41-8454-edb20c610fb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.359897 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.359899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwls\" (UniqueName: \"kubernetes.io/projected/10f31202-5e4e-4d41-8454-edb20c610fb8-kube-api-access-4qwls\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.363011 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.362975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10f31202-5e4e-4d41-8454-edb20c610fb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.368998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.368922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwls\" (UniqueName: \"kubernetes.io/projected/10f31202-5e4e-4d41-8454-edb20c610fb8-kube-api-access-4qwls\") pod \"monitoring-plugin-5d85778b9d-4fccx\" (UID: \"10f31202-5e4e-4d41-8454-edb20c610fb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.534604 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.534378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:20.739564 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.739531    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74cff45487-wt6ks"]
Sep 15 16:28:20.742571 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.742149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.745517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.745490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 16:28:20.746208 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.746188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 16:28:20.746321 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.746284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 16:28:20.746624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.746601    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 16:28:20.746624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.746616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xjntj\""
Sep 15 16:28:20.746830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.746640    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 16:28:20.751863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.751843    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 16:28:20.758955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.758881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74cff45487-wt6ks"]
Sep 15 16:28:20.864756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbql5\" (UniqueName: \"kubernetes.io/projected/8300868d-bede-4c69-89e4-e2d7abb76883-kube-api-access-nbql5\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-metrics-client-ca\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-serving-certs-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-federate-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.864935 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.864914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965511 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-metrics-client-ca\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-serving-certs-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-federate-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.965866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.965775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbql5\" (UniqueName: \"kubernetes.io/projected/8300868d-bede-4c69-89e4-e2d7abb76883-kube-api-access-nbql5\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.966657 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.966634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-serving-certs-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.966808 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.966696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-metrics-client-ca\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.967467 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.967447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.970010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.969985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-telemeter-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.970079 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.970046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.970333 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.970311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.971042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.971021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8300868d-bede-4c69-89e4-e2d7abb76883-federate-client-tls\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:20.974999 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:20.974976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbql5\" (UniqueName: \"kubernetes.io/projected/8300868d-bede-4c69-89e4-e2d7abb76883-kube-api-access-nbql5\") pod \"telemeter-client-74cff45487-wt6ks\" (UID: \"8300868d-bede-4c69-89e4-e2d7abb76883\") " pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:21.055700 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.055664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks"
Sep 15 16:28:21.687448 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.687412    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:28:21.690201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.690174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.692756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.692712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 16:28:21.693215 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.693067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 16:28:21.693306 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.693219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 16:28:21.693623 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.693601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 16:28:21.693880 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.693858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 16:28:21.698801 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.698665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kgwqw\""
Sep 15 16:28:21.711042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.711003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:28:21.733940 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.733912    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:21.744518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.744485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.750407 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.750379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 16:28:21.752200 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.751401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 16:28:21.752200 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.751649    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 16:28:21.755822 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.754728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 16:28:21.756767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 16:28:21.756767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756037    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 16:28:21.756767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756582    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 16:28:21.756767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756648    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 16:28:21.756767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jpxrj\""
Sep 15 16:28:21.757070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4b0giiks971i5\""
Sep 15 16:28:21.757070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756988    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 16:28:21.757070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.756991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 16:28:21.757225 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.757099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 16:28:21.757294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.757271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 16:28:21.758943 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.758915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 16:28:21.762096 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.762072    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 16:28:21.771727 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.771703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:21.773718 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.773838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.773838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.773838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcmk\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.773984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgzd\" (UniqueName: \"kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.774358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.774241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875605 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.875798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcmk\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.875968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgzd\" (UniqueName: \"kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.876392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876842 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876842 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.876945 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.876907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.877208 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.877089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.877298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.877229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.877371 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.877308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.879366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.878773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.880680 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.880301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.880711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.880766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.881260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.881317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.881670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.882573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.883727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.884354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.884436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.884461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.884891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.884890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.885281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.885598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.886038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.886086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:21.887864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.887713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcmk\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk\") pod \"prometheus-k8s-0\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:21.890971 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:21.890938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgzd\" (UniqueName: \"kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd\") pod \"console-68cb459b96-gbf7r\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") " pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:22.002862 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:22.002777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:22.073155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:22.073106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:23.172358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.171869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67559bd77c-77n4v"]
Sep 15 16:28:23.180511 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:23.180456    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b7a40b_3afc_4e5d_aa80_53d17c07500c.slice/crio-51264cdf6a449479833e63d67a9875ae08200c352cfb7d7fc326878eba7fe7ab WatchSource:0}: Error finding container 51264cdf6a449479833e63d67a9875ae08200c352cfb7d7fc326878eba7fe7ab: Status 404 returned error can't find the container with id 51264cdf6a449479833e63d67a9875ae08200c352cfb7d7fc326878eba7fe7ab
Sep 15 16:28:23.202392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.202352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:23.209190 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.209117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74cff45487-wt6ks"]
Sep 15 16:28:23.219033 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:28:23.219007    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8300868d_bede_4c69_89e4_e2d7abb76883.slice/crio-abab885daca80e11c505084a292b863d5ace0e813a5d498745fffba7dc7223f6 WatchSource:0}: Error finding container abab885daca80e11c505084a292b863d5ace0e813a5d498745fffba7dc7223f6: Status 404 returned error can't find the container with id abab885daca80e11c505084a292b863d5ace0e813a5d498745fffba7dc7223f6
Sep 15 16:28:23.239231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.239207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"]
Sep 15 16:28:23.252677 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.252564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c7d57d89c-b947v"]
Sep 15 16:28:23.252677 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.252661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:28:23.513134 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.513099    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627" exitCode=0
Sep 15 16:28:23.513320 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.513192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"}
Sep 15 16:28:23.513320 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.513277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"8bc319f670aa2cf16ee83d7d17544057698abd401de772d4ea7659bbdb0e4033"}
Sep 15 16:28:23.516434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.516409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts" event={"ID":"5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a","Type":"ContainerStarted","Data":"cb303cabe2984bd323e20da00417634bc82d463dde31309cea680d8bd9f5d7aa"}
Sep 15 16:28:23.519181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.519161    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="b4dd72c871fb40c28d3f0fdfae1bbaef1021dfbd756b78500619e8c495d13ebe" exitCode=0
Sep 15 16:28:23.519281 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.519236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"b4dd72c871fb40c28d3f0fdfae1bbaef1021dfbd756b78500619e8c495d13ebe"}
Sep 15 16:28:23.521063 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.520982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5g8m" event={"ID":"656cb863-0d56-4cec-838c-6f8a0bbdba5e","Type":"ContainerStarted","Data":"09743b2b6d987809449d34558a1b0214c464e42286d92fb2d9d7ced2afe2d15c"}
Sep 15 16:28:23.521125 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.521089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:28:23.522656 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.522611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tpn5j" event={"ID":"a16f1e9a-a3bb-450f-881a-15e066a3b639","Type":"ContainerStarted","Data":"6bbf12d33e398f0f0422d940dfc6cc21fb30a22c1b1fc4142047bc7fb8485430"}
Sep 15 16:28:23.524140 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.524117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx" event={"ID":"10f31202-5e4e-4d41-8454-edb20c610fb8","Type":"ContainerStarted","Data":"5d59cec866de34cc2a713e23ca5582484fcfc4fb59833aea010a0978c7dfde72"}
Sep 15 16:28:23.525895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.525866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"51264cdf6a449479833e63d67a9875ae08200c352cfb7d7fc326878eba7fe7ab"}
Sep 15 16:28:23.527436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.527388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v" event={"ID":"d0ee721d-6d7f-4143-acb1-a4c7e239b884","Type":"ContainerStarted","Data":"b4ea7815e266ff4cdab2d43166c14dc594946e5a4d5e82b4e0f9f524b34546a5"}
Sep 15 16:28:23.528889 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.528865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks" event={"ID":"8300868d-bede-4c69-89e4-e2d7abb76883","Type":"ContainerStarted","Data":"abab885daca80e11c505084a292b863d5ace0e813a5d498745fffba7dc7223f6"}
Sep 15 16:28:23.530326 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.530304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb459b96-gbf7r" event={"ID":"a6a135fc-22b3-45af-87d3-ad616ced9c82","Type":"ContainerStarted","Data":"2bfe8c06ba551ee178232c76d7c7079c94afde24b68ac13fb8ce4d8231040a32"}
Sep 15 16:28:23.533089 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.532952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2" event={"ID":"7c30c7ef-0a71-417a-b265-4265ac14fa19","Type":"ContainerStarted","Data":"d436a90c474a2931d287d5b2797c14773afafc4add50b6f47394e01f0b0c7de2"}
Sep 15 16:28:23.533089 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.533010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2" event={"ID":"7c30c7ef-0a71-417a-b265-4265ac14fa19","Type":"ContainerStarted","Data":"ddbf6d4edc3bba92eb586dc44cb1decfd411dd4c916d9e10b808ab039f533736"}
Sep 15 16:28:23.533089 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.533037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2" event={"ID":"7c30c7ef-0a71-417a-b265-4265ac14fa19","Type":"ContainerStarted","Data":"64d92595753864a0ddda2fbe95aca4f0f4fd32b401082f62de38d83f7df3c36d"}
Sep 15 16:28:23.559861 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.559803    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-psnts" podStartSLOduration=3.684639873 podStartE2EDuration="8.559781799s" totalImagesPullingTime="4.875141926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:23.558664368 +0000 UTC m=+74.945303636" watchObservedRunningTime="2026-09-15 16:28:23.559781799 +0000 UTC m=+74.946421066"
Sep 15 16:28:23.580080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.579984    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-l5sw2" podStartSLOduration=2.472109003 podStartE2EDuration="8.579966366s" totalImagesPullingTime="6.107857363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:23.577891584 +0000 UTC m=+74.964530852" watchObservedRunningTime="2026-09-15 16:28:23.579966366 +0000 UTC m=+74.966605624"
Sep 15 16:28:23.597658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.597605    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q5g8m" podStartSLOduration=67.317915773 podStartE2EDuration="1m14.597589022s" totalImagesPullingTime="7.279673249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:23.596511211 +0000 UTC m=+74.983150481" watchObservedRunningTime="2026-09-15 16:28:23.597589022 +0000 UTC m=+74.984228286"
Sep 15 16:28:23.639690 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:23.639608    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tpn5j" podStartSLOduration=67.039672704 podStartE2EDuration="1m14.639591443s" totalImagesPullingTime="7.599918739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:23.638641199 +0000 UTC m=+75.025280466" watchObservedRunningTime="2026-09-15 16:28:23.639591443 +0000 UTC m=+75.026230709"
Sep 15 16:28:37.609471 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.607144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx" event={"ID":"10f31202-5e4e-4d41-8454-edb20c610fb8","Type":"ContainerStarted","Data":"9f9b00f601e4b7db8da24666c2337d0cbbb1d8bd99c4dd64a6b1d42aee5e4487"}
Sep 15 16:28:37.609471 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.608034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:37.613823 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.613106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"98b924fc80995c4e6a077b43b7b93252950765b67ef9a4b3722081abb76ac2d0"}
Sep 15 16:28:37.615508 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.615481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v" event={"ID":"d0ee721d-6d7f-4143-acb1-a4c7e239b884","Type":"ContainerStarted","Data":"3e0f6c578bdf8a61e4291741b1df86808fcdf2691b35e77dac1b65a1615be548"}
Sep 15 16:28:37.617085 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.617025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx"
Sep 15 16:28:37.618098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.618039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks" event={"ID":"8300868d-bede-4c69-89e4-e2d7abb76883","Type":"ContainerStarted","Data":"f37673f7a7ea8e9b3114a676803bffb11823bc020c635b4105ffab7a351309b8"}
Sep 15 16:28:37.621298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.621046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb459b96-gbf7r" event={"ID":"a6a135fc-22b3-45af-87d3-ad616ced9c82","Type":"ContainerStarted","Data":"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"}
Sep 15 16:28:37.624989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.623907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-x5vt9" event={"ID":"9ca42511-83bd-42a7-a913-d80732f74f6e","Type":"ContainerStarted","Data":"8da8755bfa7298c862ee77d7274e3a31b5587b2b72cf9168da3aa73d64c08cb2"}
Sep 15 16:28:37.624989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.623974    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4fccx" podStartSLOduration=3.893724887 podStartE2EDuration="17.623959254s" totalImagesPullingTime="13.730234367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:37.623296477 +0000 UTC m=+89.009935745" watchObservedRunningTime="2026-09-15 16:28:37.623959254 +0000 UTC m=+89.010598523"
Sep 15 16:28:37.624989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.624897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:37.627363 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.627257    2578 patch_prober.go:28] interesting pod/downloads-76f99584f4-x5vt9 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body=
Sep 15 16:28:37.627363 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.627299    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-x5vt9" podUID="9ca42511-83bd-42a7-a913-d80732f74f6e" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused"
Sep 15 16:28:37.704087 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.702110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v" podStartSLOduration=4.58993335 podStartE2EDuration="18.702089658s" totalImagesPullingTime="14.112156308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:37.677107387 +0000 UTC m=+89.063746655" watchObservedRunningTime="2026-09-15 16:28:37.702089658 +0000 UTC m=+89.088728926"
Sep 15 16:28:37.704087 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.702281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68cb459b96-gbf7r" podStartSLOduration=2.589599291 podStartE2EDuration="16.702266974s" totalImagesPullingTime="14.112667683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:37.701552005 +0000 UTC m=+89.088191274" watchObservedRunningTime="2026-09-15 16:28:37.702266974 +0000 UTC m=+89.088906427"
Sep 15 16:28:37.723552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:37.722883    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-x5vt9" podStartSLOduration=1.5315860589999999 podStartE2EDuration="26.722864289s" totalImagesPullingTime="25.19127823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:37.722766769 +0000 UTC m=+89.109406039" watchObservedRunningTime="2026-09-15 16:28:37.722864289 +0000 UTC m=+89.109503557"
Sep 15 16:28:38.634452 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.634416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"}
Sep 15 16:28:38.635061 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.635037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"}
Sep 15 16:28:38.635301 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.635280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"}
Sep 15 16:28:38.635406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.635332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"}
Sep 15 16:28:38.635406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.635376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"}
Sep 15 16:28:38.635503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.635415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerStarted","Data":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"}
Sep 15 16:28:38.638591 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.638565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"4a42accf64c3b908ff10a6852480e5084f6f8039be82fb78b06834465aeddf23"}
Sep 15 16:28:38.638671 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.638629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"89cb899990f1eb32cda616b5147b7b0d2e45ad015970924524608c31796969ee"}
Sep 15 16:28:38.638751 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.638670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"e352782bcd018b69d97c39146161fa96f73d857bc3cc18ecd0e64cf42b87e31a"}
Sep 15 16:28:38.638751 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.638699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"3500197e7b93eeb5704dcfd09eac5b2d8b70d3c3d199cdefc688b9d954d299af"}
Sep 15 16:28:38.638751 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.638746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"d1c1144b1d3319123c64940515e5ee8f91eac7876a653cddbba618b3a72d08e4"}
Sep 15 16:28:38.641661 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.641631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"adf8ba9d817dbcb32448fe091a26164f23fb90e1d38d82552814024c4f0589f1"}
Sep 15 16:28:38.641784 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.641686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"496fa50407c99a24bcb9ad37998fbaeb45005dc4fdcf539ea1a17d97f3601c9e"}
Sep 15 16:28:38.644359 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.644332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks" event={"ID":"8300868d-bede-4c69-89e4-e2d7abb76883","Type":"ContainerStarted","Data":"27f3ef1827e440d3df61cae1eda4aec1228efd97b50e83a7074005737f336c97"}
Sep 15 16:28:38.644454 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.644382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks" event={"ID":"8300868d-bede-4c69-89e4-e2d7abb76883","Type":"ContainerStarted","Data":"e51dc767a450ee47f9c6c2276c6ba3b316832a85f6410f39687e9e678eee80ff"}
Sep 15 16:28:38.654240 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.654207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-x5vt9"
Sep 15 16:28:38.665622 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.665566    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.770365184 podStartE2EDuration="17.665546522s" totalImagesPullingTime="13.868175s" totalInitContainerRuntime="27.006338ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:38.662869703 +0000 UTC m=+90.049508972" watchObservedRunningTime="2026-09-15 16:28:38.665546522 +0000 UTC m=+90.052185790"
Sep 15 16:28:38.700483 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.700425    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74cff45487-wt6ks" podStartSLOduration=4.551938932 podStartE2EDuration="18.700402996s" totalImagesPullingTime="14.148464064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:38.698546497 +0000 UTC m=+90.085185766" watchObservedRunningTime="2026-09-15 16:28:38.700402996 +0000 UTC m=+90.087042264"
Sep 15 16:28:38.792381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:38.792348    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:28:40.208451 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.208365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:40.208895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.208488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:28:40.659330 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.659293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerStarted","Data":"d5330bb63c593a730b7006c377d44208b7c9f18ca0ddade115e6a05d234ac0df"}
Sep 15 16:28:40.663197 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.663166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"6c1dbefa043f73d9bdd2593abfdb07fc228d26e0667955e19f4fa30de5ea597c"}
Sep 15 16:28:40.663340 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.663236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"38244b04369fe03bc471765966f1ed82b1e247080cb1e1542b49d4ef5990713b"}
Sep 15 16:28:40.663340 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.663287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" event={"ID":"68b7a40b-3afc-4e5d-aa80-53d17c07500c","Type":"ContainerStarted","Data":"b76365f4e181c01639abd373bf42dd8a33e898b2462bd748182f43189838b7dc"}
Sep 15 16:28:40.688654 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.688603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.637466785 podStartE2EDuration="24.688582805s" totalImagesPullingTime="20.026095775s" totalInitContainerRuntime="25.020245ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:40.686067718 +0000 UTC m=+92.072706985" watchObservedRunningTime="2026-09-15 16:28:40.688582805 +0000 UTC m=+92.075222074"
Sep 15 16:28:40.710193 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:40.710140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v" podStartSLOduration=6.963009563 podStartE2EDuration="22.710117822s" totalImagesPullingTime="15.747108259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:40.70831365 +0000 UTC m=+92.094952941" watchObservedRunningTime="2026-09-15 16:28:40.710117822 +0000 UTC m=+92.096757089"
Sep 15 16:28:41.668800 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:41.668767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:42.003487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:42.003404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:28:42.073620 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:42.073582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:42.680141 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:42.680104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67559bd77c-77n4v"
Sep 15 16:28:54.540031 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:28:54.539998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q5g8m"
Sep 15 16:29:00.213594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:00.213562    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:29:00.217416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:00.217394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c7d57d89c-b947v"
Sep 15 16:29:04.674870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.674806    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68cb459b96-gbf7r" podUID="a6a135fc-22b3-45af-87d3-ad616ced9c82" containerName="console" containerID="cri-o://803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360" gracePeriod=15
Sep 15 16:29:04.915259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.915236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:29:04.974895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.974803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.974895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.974850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.975110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.974929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.975110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.974956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wgzd\" (UniqueName: \"kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.975110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.974988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.975110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.975024    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config\") pod \"a6a135fc-22b3-45af-87d3-ad616ced9c82\" (UID: \"a6a135fc-22b3-45af-87d3-ad616ced9c82\") "
Sep 15 16:29:04.975347 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.975311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:04.975425 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.975401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca" (OuterVolumeSpecName: "service-ca") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:04.975524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.975493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config" (OuterVolumeSpecName: "console-config") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:04.977387 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.977357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:04.977387 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.977365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd" (OuterVolumeSpecName: "kube-api-access-4wgzd") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "kube-api-access-4wgzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:04.977530 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:04.977407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a6a135fc-22b3-45af-87d3-ad616ced9c82" (UID: "a6a135fc-22b3-45af-87d3-ad616ced9c82"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:05.075666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075627    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-serving-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.075666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075658    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-service-ca\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.075666 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075671    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-oauth-serving-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.075955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgzd\" (UniqueName: \"kubernetes.io/projected/a6a135fc-22b3-45af-87d3-ad616ced9c82-kube-api-access-4wgzd\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.075955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075695    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-oauth-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.075955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.075708    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6a135fc-22b3-45af-87d3-ad616ced9c82-console-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:05.775486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.775444    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6a135fc-22b3-45af-87d3-ad616ced9c82" containerID="803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360" exitCode=0
Sep 15 16:29:05.775486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.775489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb459b96-gbf7r"
Sep 15 16:29:05.776130 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.775501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb459b96-gbf7r" event={"ID":"a6a135fc-22b3-45af-87d3-ad616ced9c82","Type":"ContainerDied","Data":"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"}
Sep 15 16:29:05.776130 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.775549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb459b96-gbf7r" event={"ID":"a6a135fc-22b3-45af-87d3-ad616ced9c82","Type":"ContainerDied","Data":"2bfe8c06ba551ee178232c76d7c7079c94afde24b68ac13fb8ce4d8231040a32"}
Sep 15 16:29:05.776130 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.775583    2578 scope.go:117] "RemoveContainer" containerID="803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"
Sep 15 16:29:05.783857 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.783839    2578 scope.go:117] "RemoveContainer" containerID="803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"
Sep 15 16:29:05.784170 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:05.784149    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360\": container with ID starting with 803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360 not found: ID does not exist" containerID="803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"
Sep 15 16:29:05.784219 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.784180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360"} err="failed to get container status \"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360\": rpc error: code = NotFound desc = could not find container \"803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360\": container with ID starting with 803cd8c69522288623f89ff9f44fa9b779ebfe1e0680ac6fe6360211ad4c1360 not found: ID does not exist"
Sep 15 16:29:05.795765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.792758    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:29:05.795890 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:05.795848    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68cb459b96-gbf7r"]
Sep 15 16:29:07.153590 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:07.153553    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a135fc-22b3-45af-87d3-ad616ced9c82" path="/var/lib/kubelet/pods/a6a135fc-22b3-45af-87d3-ad616ced9c82/volumes"
Sep 15 16:29:22.073792 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:22.073729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:22.134651 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:22.134624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:22.868235 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:22.868204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:35.753932 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.753892    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:35.754438 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754208    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="alertmanager" containerID="cri-o://4a42accf64c3b908ff10a6852480e5084f6f8039be82fb78b06834465aeddf23" gracePeriod=120
Sep 15 16:29:35.754438 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754329    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="prom-label-proxy" containerID="cri-o://d5330bb63c593a730b7006c377d44208b7c9f18ca0ddade115e6a05d234ac0df" gracePeriod=120
Sep 15 16:29:35.754438 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754350    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-web" containerID="cri-o://3500197e7b93eeb5704dcfd09eac5b2d8b70d3c3d199cdefc688b9d954d299af" gracePeriod=120
Sep 15 16:29:35.754438 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754363    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="config-reloader" containerID="cri-o://d1c1144b1d3319123c64940515e5ee8f91eac7876a653cddbba618b3a72d08e4" gracePeriod=120
Sep 15 16:29:35.754687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754360    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy" containerID="cri-o://e352782bcd018b69d97c39146161fa96f73d857bc3cc18ecd0e64cf42b87e31a" gracePeriod=120
Sep 15 16:29:35.754687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.754285    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-metric" containerID="cri-o://89cb899990f1eb32cda616b5147b7b0d2e45ad015970924524608c31796969ee" gracePeriod=120
Sep 15 16:29:35.904774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904720    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="d5330bb63c593a730b7006c377d44208b7c9f18ca0ddade115e6a05d234ac0df" exitCode=0
Sep 15 16:29:35.904774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904768    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="e352782bcd018b69d97c39146161fa96f73d857bc3cc18ecd0e64cf42b87e31a" exitCode=0
Sep 15 16:29:35.904774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904776    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="d1c1144b1d3319123c64940515e5ee8f91eac7876a653cddbba618b3a72d08e4" exitCode=0
Sep 15 16:29:35.904951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904783    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="4a42accf64c3b908ff10a6852480e5084f6f8039be82fb78b06834465aeddf23" exitCode=0
Sep 15 16:29:35.904951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"d5330bb63c593a730b7006c377d44208b7c9f18ca0ddade115e6a05d234ac0df"}
Sep 15 16:29:35.904951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"e352782bcd018b69d97c39146161fa96f73d857bc3cc18ecd0e64cf42b87e31a"}
Sep 15 16:29:35.904951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"d1c1144b1d3319123c64940515e5ee8f91eac7876a653cddbba618b3a72d08e4"}
Sep 15 16:29:35.904951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:35.904898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"4a42accf64c3b908ff10a6852480e5084f6f8039be82fb78b06834465aeddf23"}
Sep 15 16:29:36.913273 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:36.913236    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="89cb899990f1eb32cda616b5147b7b0d2e45ad015970924524608c31796969ee" exitCode=0
Sep 15 16:29:36.913273 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:36.913272    2578 generic.go:356] "Generic (PLEG): container finished" podID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerID="3500197e7b93eeb5704dcfd09eac5b2d8b70d3c3d199cdefc688b9d954d299af" exitCode=0
Sep 15 16:29:36.913673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:36.913353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"89cb899990f1eb32cda616b5147b7b0d2e45ad015970924524608c31796969ee"}
Sep 15 16:29:36.913673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:36.913420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"3500197e7b93eeb5704dcfd09eac5b2d8b70d3c3d199cdefc688b9d954d299af"}
Sep 15 16:29:37.002777 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.002750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:37.062348 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sthp\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.062523 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.062523 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.062640 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.062640 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062567    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.062640 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy\") pod \"074af8b5-cbdf-41b5-bf28-a39d63440c15\" (UID: \"074af8b5-cbdf-41b5-bf28-a39d63440c15\") "
Sep 15 16:29:37.063083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.062927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:37.063531 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.063138    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.066083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.065948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:37.066370 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.066335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.066461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.066377    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:37.066527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.066343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp" (OuterVolumeSpecName: "kube-api-access-7sthp") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "kube-api-access-7sthp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:37.066676 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.066639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.066910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.066877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.067695 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.067673    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume" (OuterVolumeSpecName: "config-volume") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.067903 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.067877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.068189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.068150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out" (OuterVolumeSpecName: "config-out") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:37.068439 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.068417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:37.071787 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.071684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.080367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.080343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config" (OuterVolumeSpecName: "web-config") pod "074af8b5-cbdf-41b5-bf28-a39d63440c15" (UID: "074af8b5-cbdf-41b5-bf28-a39d63440c15"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:37.164409 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164385    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sthp\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-kube-api-access-7sthp\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164411    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164423    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-cluster-tls-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164431    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164440    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-main-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164448    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-out\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164457    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/074af8b5-cbdf-41b5-bf28-a39d63440c15-tls-assets\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164465    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-config-volume\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164474    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/074af8b5-cbdf-41b5-bf28-a39d63440c15-alertmanager-main-db\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164482    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-web-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164491    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/074af8b5-cbdf-41b5-bf28-a39d63440c15-metrics-client-ca\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.164509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.164499    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/074af8b5-cbdf-41b5-bf28-a39d63440c15-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:37.920879 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.920844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"074af8b5-cbdf-41b5-bf28-a39d63440c15","Type":"ContainerDied","Data":"b1b763af521d95e28a310f9a3a93a5a9e364771dcdd162db9a68296d0e6f36d0"}
Sep 15 16:29:37.921243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.920910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:37.921243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.920921    2578 scope.go:117] "RemoveContainer" containerID="d5330bb63c593a730b7006c377d44208b7c9f18ca0ddade115e6a05d234ac0df"
Sep 15 16:29:37.928515 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.928497    2578 scope.go:117] "RemoveContainer" containerID="89cb899990f1eb32cda616b5147b7b0d2e45ad015970924524608c31796969ee"
Sep 15 16:29:37.935727 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.935705    2578 scope.go:117] "RemoveContainer" containerID="e352782bcd018b69d97c39146161fa96f73d857bc3cc18ecd0e64cf42b87e31a"
Sep 15 16:29:37.941183 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.941162    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:37.943660 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.943641    2578 scope.go:117] "RemoveContainer" containerID="3500197e7b93eeb5704dcfd09eac5b2d8b70d3c3d199cdefc688b9d954d299af"
Sep 15 16:29:37.946546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.946524    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:37.951243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.951226    2578 scope.go:117] "RemoveContainer" containerID="d1c1144b1d3319123c64940515e5ee8f91eac7876a653cddbba618b3a72d08e4"
Sep 15 16:29:37.958197 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.958181    2578 scope.go:117] "RemoveContainer" containerID="4a42accf64c3b908ff10a6852480e5084f6f8039be82fb78b06834465aeddf23"
Sep 15 16:29:37.965251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.965229    2578 scope.go:117] "RemoveContainer" containerID="b4dd72c871fb40c28d3f0fdfae1bbaef1021dfbd756b78500619e8c495d13ebe"
Sep 15 16:29:37.975987 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.975963    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:37.976407 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976390    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-web"
Sep 15 16:29:37.976407 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976411    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-web"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976424    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-metric"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976433    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-metric"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976445    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="init-config-reloader"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976454    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="init-config-reloader"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976467    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a135fc-22b3-45af-87d3-ad616ced9c82" containerName="console"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976474    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a135fc-22b3-45af-87d3-ad616ced9c82" containerName="console"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976499    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="prom-label-proxy"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976507    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="prom-label-proxy"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976522    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976530    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="alertmanager"
Sep 15 16:29:37.976546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="alertmanager"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="config-reloader"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="config-reloader"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976630    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976641    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="prom-label-proxy"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976650    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-web"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976660    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a135fc-22b3-45af-87d3-ad616ced9c82" containerName="console"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="alertmanager"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976678    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="config-reloader"
Sep 15 16:29:37.977069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.976686    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" containerName="kube-rbac-proxy-metric"
Sep 15 16:29:37.995184 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.995133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:37.995336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:37.995295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.008368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.008496    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.008593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.008869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.009035    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.009422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bjrpj\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.009517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 16:29:38.010813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.009986    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 16:29:38.021484 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.021455    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 16:29:38.027461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.027435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 16:29:38.072277 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-web-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-config-volume\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-config-out\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-kube-api-access-ftvmj\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.072688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.072630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.173951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.173903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.173951 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.173959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-web-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.173977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.173996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-config-volume\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-config-out\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-kube-api-access-ftvmj\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.174678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.174603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.175170 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.175142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.175535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.175512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a46d88f4-70fd-4aef-854b-34667d405528-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.177249 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.177221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.177249 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.177242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-web-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.177416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.177286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.177416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.177374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-config-volume\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.177416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.177400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.178346 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.178319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.178636 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.178617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.179097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.179069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a46d88f4-70fd-4aef-854b-34667d405528-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.179266 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.179248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a46d88f4-70fd-4aef-854b-34667d405528-config-out\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.184248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.184226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/a46d88f4-70fd-4aef-854b-34667d405528-kube-api-access-ftvmj\") pod \"alertmanager-main-0\" (UID: \"a46d88f4-70fd-4aef-854b-34667d405528\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.334349 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.334316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 16:29:38.615874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.615850    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 16:29:38.617896 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:29:38.617871    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46d88f4_70fd_4aef_854b_34667d405528.slice/crio-f4efe8aea224b658286c07affc0d1ae04d081912354d79af2103c2599c032f01 WatchSource:0}: Error finding container f4efe8aea224b658286c07affc0d1ae04d081912354d79af2103c2599c032f01: Status 404 returned error can't find the container with id f4efe8aea224b658286c07affc0d1ae04d081912354d79af2103c2599c032f01
Sep 15 16:29:38.927555 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.927524    2578 generic.go:356] "Generic (PLEG): container finished" podID="a46d88f4-70fd-4aef-854b-34667d405528" containerID="1c19c28b99d1dba796ffa036e89fc0ca7f141f32a5f2fa1e4197cab1b7d1bb4a" exitCode=0
Sep 15 16:29:38.927938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.927596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerDied","Data":"1c19c28b99d1dba796ffa036e89fc0ca7f141f32a5f2fa1e4197cab1b7d1bb4a"}
Sep 15 16:29:38.927938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:38.927643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"f4efe8aea224b658286c07affc0d1ae04d081912354d79af2103c2599c032f01"}
Sep 15 16:29:39.154651 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.154624    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="074af8b5-cbdf-41b5-bf28-a39d63440c15" path="/var/lib/kubelet/pods/074af8b5-cbdf-41b5-bf28-a39d63440c15/volumes"
Sep 15 16:29:39.935486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"76f3a37743c1cedf68e9309b496eb071a4792a2ac1daa41521178d5d34b334ed"}
Sep 15 16:29:39.935894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"dcdd7f3f453bf5f94c13fa81c9f868dbcd1131f842e0ebb188bb53fdd0095c26"}
Sep 15 16:29:39.935894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"447a1b107c9a3d35056941d0e30b0572686c0ac87d90a9506e77de7b38766981"}
Sep 15 16:29:39.935894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"4412c7cf62845bd1b9ee8d42f38606e0a57ac528cbff3b45db1f37ebd3e5f9fc"}
Sep 15 16:29:39.935894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"a23e04cad6454367345b1b2075131e50558181758213a4896dc761fb3c9c3bd5"}
Sep 15 16:29:39.935894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.935633    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a46d88f4-70fd-4aef-854b-34667d405528","Type":"ContainerStarted","Data":"6e53c6d2e3923157f4ee5c7134d1155dbefd37da9160486f0ca882470440abd7"}
Sep 15 16:29:39.961934 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:39.961876    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.944853518 podStartE2EDuration="2.961858167s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.004649ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:39.96078688 +0000 UTC m=+151.347426173" watchObservedRunningTime="2026-09-15 16:29:39.961858167 +0000 UTC m=+151.348497435"
Sep 15 16:29:40.008231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.007329    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:40.008231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.007682    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="prometheus" containerID="cri-o://b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5" gracePeriod=600
Sep 15 16:29:40.008231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.008092    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0" gracePeriod=600
Sep 15 16:29:40.008231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.008230    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy" containerID="cri-o://2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489" gracePeriod=600
Sep 15 16:29:40.008599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.008311    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-web" containerID="cri-o://aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be" gracePeriod=600
Sep 15 16:29:40.008599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.008380    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="thanos-sidecar" containerID="cri-o://ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194" gracePeriod=600
Sep 15 16:29:40.008599 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.008446    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="config-reloader" containerID="cri-o://608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697" gracePeriod=600
Sep 15 16:29:40.259369 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.259067    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwcmk\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.294839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.294835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.296833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.296796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1\") pod \"88d10be7-812a-442b-bd83-574233e568ad\" (UID: \"88d10be7-812a-442b-bd83-574233e568ad\") "
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297916    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297936    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297953    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297968    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.298216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.297983    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-db\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.299514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.298654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.299514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.299057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.300195 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.300148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.300945 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.300915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:40.301113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.301039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config" (OuterVolumeSpecName: "config") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.302303 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.302278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.303136 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.303106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out" (OuterVolumeSpecName: "config-out") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:40.303136 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.303109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:40.307238 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.307125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.307238 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.307312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk" (OuterVolumeSpecName: "kube-api-access-dwcmk") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "kube-api-access-dwcmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:40.307501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.307364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.308033 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.307999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.309205 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.309174    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.311294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.311265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.315192 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.315171    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config" (OuterVolumeSpecName: "web-config") pod "88d10be7-812a-442b-bd83-574233e568ad" (UID: "88d10be7-812a-442b-bd83-574233e568ad"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:40.399232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399202    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-configmap-metrics-client-ca\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399229    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399241    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-kube-rbac-proxy\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399251    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399260    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399269    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-grpc-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399277    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399285    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/88d10be7-812a-442b-bd83-574233e568ad-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399294    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399304    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399313    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-web-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399321    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwcmk\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-kube-api-access-dwcmk\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399328    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88d10be7-812a-442b-bd83-574233e568ad-config-out\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399337    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88d10be7-812a-442b-bd83-574233e568ad-tls-assets\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.399429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.399345    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/88d10be7-812a-442b-bd83-574233e568ad-secret-metrics-client-certs\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:29:40.942750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942704    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0" exitCode=0
Sep 15 16:29:40.942750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942730    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489" exitCode=0
Sep 15 16:29:40.942750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942751    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be" exitCode=0
Sep 15 16:29:40.942750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942756    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194" exitCode=0
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942763    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697" exitCode=0
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942768    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d10be7-812a-442b-bd83-574233e568ad" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5" exitCode=0
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"}
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942911    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:40.943336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.942963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"88d10be7-812a-442b-bd83-574233e568ad","Type":"ContainerDied","Data":"8bc319f670aa2cf16ee83d7d17544057698abd401de772d4ea7659bbdb0e4033"}
Sep 15 16:29:40.951455 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.951433    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:40.958784 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.958726    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:40.966392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.966365    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:40.968056 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.967676    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:40.972182 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.972163    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:40.975592 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.975566    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:40.982560 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.982545    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:40.990068 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.990049    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:40.995491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995470    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:40.995780 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="thanos-sidecar"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="thanos-sidecar"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995802    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995807    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995814    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="prometheus"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995820    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="prometheus"
Sep 15 16:29:40.995829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="config-reloader"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="config-reloader"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="init-config-reloader"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995845    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="init-config-reloader"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995855    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-web"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-web"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995865    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995870    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995913    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995922    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="config-reloader"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995927    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="thanos-sidecar"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995935    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="prometheus"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995941    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy"
Sep 15 16:29:40.996015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.995948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d10be7-812a-442b-bd83-574233e568ad" containerName="kube-rbac-proxy-web"
Sep 15 16:29:40.997562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.997544    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:40.997960 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.997877    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:40.997960 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.997922    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:40.997960 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.997947    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:40.998292 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.998275    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:40.998366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998301    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:40.998366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998323    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:40.998576 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.998558    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:40.998618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:40.998618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998597    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:40.998934 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.998911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:40.998999 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998939    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:40.998999 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.998959    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:40.999244 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.999227    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:40.999285 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.999249    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:40.999285 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.999262    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:40.999531 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.999512    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:40.999588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.999536    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:40.999588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.999550    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.000005 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:29:40.999967    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.000005 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:40.999997    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.000160 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000017    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:41.000160 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.000694 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:41.000694 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000282    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:41.000876 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:41.000876 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.000787    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:41.001232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001138    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:41.001232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001166    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:41.001404 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001385    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:41.001469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001405    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:41.001680 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:41.001779 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.001681    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:41.002871 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.002842    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:41.002871 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.002867    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.003021 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.002880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4b0giiks971i5\""
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003127    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003315    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003342    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 16:29:41.003524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003694    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jpxrj\""
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003836    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003859    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 16:29:41.003956 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003939    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 16:29:41.004287 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.003979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 16:29:41.004478 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.004461    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 16:29:41.004600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.004517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 16:29:41.004873 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.004772    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:41.004873 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.004801    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:41.005251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005156    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:41.005251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005184    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:41.005562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005542    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:41.005643 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005563    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:41.005900 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005875    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:41.006643 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.005902    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.007114    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.007143    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.007845    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.007872    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.009440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 16:29:41.010757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.010469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 16:29:41.012576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.012548    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:41.012576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.012572    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:41.012894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.012874    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:41.012962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.012895    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:41.013129 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013111    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:41.013180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013130    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:41.013347 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013327    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:41.013400 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013349    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:41.013672 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013645    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:41.013796 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013674    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:41.014004 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.013978    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:41.014069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014007    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.014275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014257    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.014317 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014276    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:41.014506 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014490    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:41.014559 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014506    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:41.014718 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014701    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:41.014795 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014718    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:41.014965 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:41.015022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.014967    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:41.015248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015222    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:41.015326 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015249    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:41.015509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015488    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:41.015564 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015511    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:41.015782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015754    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:41.015863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.015783    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.016040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016023    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.016100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016040    2578 scope.go:117] "RemoveContainer" containerID="2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"
Sep 15 16:29:41.016276 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016258    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0"} err="failed to get container status \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": rpc error: code = NotFound desc = could not find container \"2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0\": container with ID starting with 2ffa17ea34c0280ebdef96c0eb450d3f3dc69d3720c6ebfbef11e8b9b27357d0 not found: ID does not exist"
Sep 15 16:29:41.016328 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016277    2578 scope.go:117] "RemoveContainer" containerID="2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"
Sep 15 16:29:41.016474 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016459    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489"} err="failed to get container status \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": rpc error: code = NotFound desc = could not find container \"2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489\": container with ID starting with 2662fcf9f4249d27ce2ccdaf1643dd3e098121ccbd110ebacceb6d9997c0e489 not found: ID does not exist"
Sep 15 16:29:41.016526 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016476    2578 scope.go:117] "RemoveContainer" containerID="aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"
Sep 15 16:29:41.016682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016666    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be"} err="failed to get container status \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": rpc error: code = NotFound desc = could not find container \"aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be\": container with ID starting with aa608cd6a16de003d569007e0a15d4964e4fd835cff59985f65a00c5f225d9be not found: ID does not exist"
Sep 15 16:29:41.016726 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016682    2578 scope.go:117] "RemoveContainer" containerID="ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"
Sep 15 16:29:41.016943 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194"} err="failed to get container status \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": rpc error: code = NotFound desc = could not find container \"ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194\": container with ID starting with ba178e777a76b483772cf325fa284574ab79e380c9109e021d2b705128a85194 not found: ID does not exist"
Sep 15 16:29:41.016985 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.016946    2578 scope.go:117] "RemoveContainer" containerID="608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"
Sep 15 16:29:41.017173 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.017141    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697"} err="failed to get container status \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": rpc error: code = NotFound desc = could not find container \"608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697\": container with ID starting with 608cfffdd8356965cc2731b271f82f1ef1fb34294f91d8a5478c5757abdaa697 not found: ID does not exist"
Sep 15 16:29:41.017173 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.017163    2578 scope.go:117] "RemoveContainer" containerID="b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"
Sep 15 16:29:41.017406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.017386    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5"} err="failed to get container status \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": rpc error: code = NotFound desc = could not find container \"b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5\": container with ID starting with b1b472aa9edc26cb85a001398270a77e4c5cd15a781a31d55cbc08d38d2572d5 not found: ID does not exist"
Sep 15 16:29:41.017459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.017408    2578 scope.go:117] "RemoveContainer" containerID="b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"
Sep 15 16:29:41.017644 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.017625    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627"} err="failed to get container status \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": rpc error: code = NotFound desc = could not find container \"b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627\": container with ID starting with b3fa9c7d67191f7e35baba79811954498ffd87dc5eac44eba7195c873f92c627 not found: ID does not exist"
Sep 15 16:29:41.021264 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.021244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:41.103771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.103973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.103999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd78q\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-kube-api-access-qd78q\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104409 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.104409 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.104204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.153455 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.153411    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d10be7-812a-442b-bd83-574233e568ad" path="/var/lib/kubelet/pods/88d10be7-812a-442b-bd83-574233e568ad/volumes"
Sep 15 16:29:41.204663 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204837 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204837 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204837 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.204998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.204973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.205263 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.205970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.206030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.206077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd78q\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-kube-api-access-qd78q\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.206109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.206178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.206140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.207070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.206911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.207699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.207671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config-out\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208021 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.207998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208105 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.208046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-web-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.208101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.208148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208550 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.208526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.208756 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.208680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.209065 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.209041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.209211 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.209092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.209281 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.209191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.209771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.209723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.210406 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.210380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.210851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.210833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.211211 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.211193    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.211594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.211575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-config\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.212019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.212000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.216225 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.216206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd78q\" (UniqueName: \"kubernetes.io/projected/5e20c9c2-fa7a-4d58-8061-33aedda36fb7-kube-api-access-qd78q\") pod \"prometheus-k8s-0\" (UID: \"5e20c9c2-fa7a-4d58-8061-33aedda36fb7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.318043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.317987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:41.595556 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.595522    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:41.598586 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:29:41.598559    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e20c9c2_fa7a_4d58_8061_33aedda36fb7.slice/crio-0b5759da6221dae91679a2085b9dd4e484e7bc89f13652293c5a14d7a4e0e914 WatchSource:0}: Error finding container 0b5759da6221dae91679a2085b9dd4e484e7bc89f13652293c5a14d7a4e0e914: Status 404 returned error can't find the container with id 0b5759da6221dae91679a2085b9dd4e484e7bc89f13652293c5a14d7a4e0e914
Sep 15 16:29:41.949637 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.949607    2578 generic.go:356] "Generic (PLEG): container finished" podID="5e20c9c2-fa7a-4d58-8061-33aedda36fb7" containerID="d707abc3dbf570a5a11acb03ad2a01dc8a97453724ebd20f47f45170a419427b" exitCode=0
Sep 15 16:29:41.950045 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.949662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerDied","Data":"d707abc3dbf570a5a11acb03ad2a01dc8a97453724ebd20f47f45170a419427b"}
Sep 15 16:29:41.950045 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:41.949709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"0b5759da6221dae91679a2085b9dd4e484e7bc89f13652293c5a14d7a4e0e914"}
Sep 15 16:29:42.956801 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"52c66aefad6275689d2c1f84e2be7c5d09af21c7f76a4259eb4e5b929008eb44"}
Sep 15 16:29:42.957217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"2f16dfba1523b68ffd2c756dee2ab54c01bf26365d223685f825edc99fe25e75"}
Sep 15 16:29:42.957217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"d96790e38605710d65b6bdb03919ad64470319b616a28f757337c75209e3752f"}
Sep 15 16:29:42.957217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"9619f2f8ef4300570573c19b77f650d3b5da02e0092ee1d1b5ee42968a4d27c4"}
Sep 15 16:29:42.957217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"b605e89baa3ae1132a5923d157302ccafb09108af2958bb4627ac9d0579186b2"}
Sep 15 16:29:42.957217 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.956955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5e20c9c2-fa7a-4d58-8061-33aedda36fb7","Type":"ContainerStarted","Data":"2c64fb4007a87c1e3fbbd943a5471edf9de8320a1d3b1178d9801c2f76f1aa44"}
Sep 15 16:29:42.984416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:42.984369    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.960322747 podStartE2EDuration="2.984346495s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.023748ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:42.982079662 +0000 UTC m=+154.368718942" watchObservedRunningTime="2026-09-15 16:29:42.984346495 +0000 UTC m=+154.370985816"
Sep 15 16:29:46.319105 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:29:46.319067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:41.319151 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:30:41.319119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:41.334980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:30:41.334953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:42.205418 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:30:42.205385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:32:09.034615 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:32:09.034580    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 16:34:01.281154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.281120    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-vmf5j"]
Sep 15 16:34:01.283595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.283573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.286098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.286074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 15 16:34:01.286206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.286078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 15 16:34:01.286953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.286936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-w6m2f\""
Sep 15 16:34:01.286953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.286946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 15 16:34:01.287090 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.286980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 15 16:34:01.293986 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.293966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vmf5j"]
Sep 15 16:34:01.313522 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.313489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.313634 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.313563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkm24\" (UniqueName: \"kubernetes.io/projected/a5983d46-8f2f-4cb6-bebe-3c3433a60148-kube-api-access-qkm24\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.313634 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.313609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.414693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.414659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.414921 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.414708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkm24\" (UniqueName: \"kubernetes.io/projected/a5983d46-8f2f-4cb6-bebe-3c3433a60148-kube-api-access-qkm24\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.414921 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.414762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.414921 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:34:01.414841    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 15 16:34:01.414921 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:34:01.414853    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 15 16:34:01.414921 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:34:01.414920    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs podName:a5983d46-8f2f-4cb6-bebe-3c3433a60148 nodeName:}" failed. No retries permitted until 2026-09-15 16:34:01.914899863 +0000 UTC m=+413.301539115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs") pod "odh-model-controller-7b966d7984-vmf5j" (UID: "a5983d46-8f2f-4cb6-bebe-3c3433a60148") : secret "odh-model-controller-metrics-tls" not found
Sep 15 16:34:01.415120 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:34:01.414938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert podName:a5983d46-8f2f-4cb6-bebe-3c3433a60148 nodeName:}" failed. No retries permitted until 2026-09-15 16:34:01.914929707 +0000 UTC m=+413.301568954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert") pod "odh-model-controller-7b966d7984-vmf5j" (UID: "a5983d46-8f2f-4cb6-bebe-3c3433a60148") : secret "odh-model-controller-webhook-cert" not found
Sep 15 16:34:01.424687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.424663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkm24\" (UniqueName: \"kubernetes.io/projected/a5983d46-8f2f-4cb6-bebe-3c3433a60148-kube-api-access-qkm24\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.918177 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.918142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.918349 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.918208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.920856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.920828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-cert\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:01.920973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:01.920896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5983d46-8f2f-4cb6-bebe-3c3433a60148-metrics-certs\") pod \"odh-model-controller-7b966d7984-vmf5j\" (UID: \"a5983d46-8f2f-4cb6-bebe-3c3433a60148\") " pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:02.194982 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:02.194901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:02.479251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:02.479171    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vmf5j"]
Sep 15 16:34:02.482564 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:34:02.482538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5983d46_8f2f_4cb6_bebe_3c3433a60148.slice/crio-d31eabbb95e8821a9ecf138b1ad562f75b3fae232612bdb88bdacf0204018111 WatchSource:0}: Error finding container d31eabbb95e8821a9ecf138b1ad562f75b3fae232612bdb88bdacf0204018111: Status 404 returned error can't find the container with id d31eabbb95e8821a9ecf138b1ad562f75b3fae232612bdb88bdacf0204018111
Sep 15 16:34:02.484058 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:02.484039    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:34:02.981497 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:02.981461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vmf5j" event={"ID":"a5983d46-8f2f-4cb6-bebe-3c3433a60148","Type":"ContainerStarted","Data":"d31eabbb95e8821a9ecf138b1ad562f75b3fae232612bdb88bdacf0204018111"}
Sep 15 16:34:05.997331 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:05.997296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vmf5j" event={"ID":"a5983d46-8f2f-4cb6-bebe-3c3433a60148","Type":"ContainerStarted","Data":"0e4dbab96731c93b80cb7c3019ced88b0544abb4e62a522e24d8e1aa5c5ebfa8"}
Sep 15 16:34:05.997771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:05.997398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:06.013461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:06.013401    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-vmf5j" podStartSLOduration=2.457154507 podStartE2EDuration="5.013386487s" totalImagesPullingTime="2.55623198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:06.012547239 +0000 UTC m=+417.399186506" watchObservedRunningTime="2026-09-15 16:34:06.013386487 +0000 UTC m=+417.400025798"
Sep 15 16:34:17.003658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.003624    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-vmf5j"
Sep 15 16:34:17.917813 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.917780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-g4db8"]
Sep 15 16:34:17.920471 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.920453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g4db8"
Sep 15 16:34:17.922786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.922757    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 15 16:34:17.922920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.922900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d4bf6\""
Sep 15 16:34:17.922990 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.922920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 15 16:34:17.928988 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:17.928965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g4db8"]
Sep 15 16:34:18.054975 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.054940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.055350 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.055024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bzx\" (UniqueName: \"kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.156343 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.156304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.156503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.156390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bzx\" (UniqueName: \"kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.156793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.156772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.168972 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.168911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bzx\" (UniqueName: \"kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx\") pod \"s3-init-g4db8\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") " pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.239806 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.239774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g4db8"
Sep 15 16:34:18.515798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:18.515703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g4db8"]
Sep 15 16:34:18.520035 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:34:18.519990    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1db47f0_f55c_4445_a648_32d814b3f1e3.slice/crio-579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13 WatchSource:0}: Error finding container 579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13: Status 404 returned error can't find the container with id 579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13
Sep 15 16:34:19.057431 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:19.057387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerStarted","Data":"579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13"}
Sep 15 16:34:23.078689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:23.078658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerStarted","Data":"678ae0a6fa80bfe81e10ccbb029abdabb4d3ec76e11535b9273b42b4ce8be636"}
Sep 15 16:34:25.088668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:25.088635    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerID="678ae0a6fa80bfe81e10ccbb029abdabb4d3ec76e11535b9273b42b4ce8be636" exitCode=0
Sep 15 16:34:25.089065 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:25.088695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerDied","Data":"678ae0a6fa80bfe81e10ccbb029abdabb4d3ec76e11535b9273b42b4ce8be636"}
Sep 15 16:34:30.114507 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:30.114466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerStarted","Data":"7e8b5cc70c4539fdc851d3bbe444ff22fa131ffdd1056ada3962c7dcb1f85998"}
Sep 15 16:34:30.146425 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:34:30.146370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-g4db8" podStartSLOduration=2.720859037 podStartE2EDuration="13.14635334s" totalImagesPullingTime="8.517990823s" totalInitContainerRuntime="1.90750348s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:30.14626499 +0000 UTC m=+441.532904257" watchObservedRunningTime="2026-09-15 16:34:30.14635334 +0000 UTC m=+441.532992607"
Sep 15 16:35:34.374486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:34.374451    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerID="7e8b5cc70c4539fdc851d3bbe444ff22fa131ffdd1056ada3962c7dcb1f85998" exitCode=0
Sep 15 16:35:34.374866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:34.374517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerDied","Data":"7e8b5cc70c4539fdc851d3bbe444ff22fa131ffdd1056ada3962c7dcb1f85998"}
Sep 15 16:35:35.502195 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.502170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g4db8"
Sep 15 16:35:35.518804 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.518729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache\") pod \"c1db47f0-f55c-4445-a648-32d814b3f1e3\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") "
Sep 15 16:35:35.518928 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.518844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5bzx\" (UniqueName: \"kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx\") pod \"c1db47f0-f55c-4445-a648-32d814b3f1e3\" (UID: \"c1db47f0-f55c-4445-a648-32d814b3f1e3\") "
Sep 15 16:35:35.521703 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.521672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx" (OuterVolumeSpecName: "kube-api-access-z5bzx") pod "c1db47f0-f55c-4445-a648-32d814b3f1e3" (UID: "c1db47f0-f55c-4445-a648-32d814b3f1e3"). InnerVolumeSpecName "kube-api-access-z5bzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:35:35.538867 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.538820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1db47f0-f55c-4445-a648-32d814b3f1e3" (UID: "c1db47f0-f55c-4445-a648-32d814b3f1e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:35:35.620414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.620387    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1db47f0-f55c-4445-a648-32d814b3f1e3-model-cache\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:35:35.620414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:35.620409    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bzx\" (UniqueName: \"kubernetes.io/projected/c1db47f0-f55c-4445-a648-32d814b3f1e3-kube-api-access-z5bzx\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:35:36.384348 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:36.384313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g4db8" event={"ID":"c1db47f0-f55c-4445-a648-32d814b3f1e3","Type":"ContainerDied","Data":"579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13"}
Sep 15 16:35:36.384348 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:36.384335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g4db8"
Sep 15 16:35:36.384348 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:36.384356    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="579d1274f6dc04a170eb92d3d0df520535bdb245799af3c999a9a731b5afed13"
Sep 15 16:35:43.866767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.866718    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-szxbz"]
Sep 15 16:35:43.867246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.867071    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerName="download-hf-model"
Sep 15 16:35:43.867246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.867083    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerName="download-hf-model"
Sep 15 16:35:43.867246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.867105    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerName="s3-init"
Sep 15 16:35:43.867246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.867110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerName="s3-init"
Sep 15 16:35:43.867246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.867160    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1db47f0-f55c-4445-a648-32d814b3f1e3" containerName="s3-init"
Sep 15 16:35:43.869337 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.869320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.871658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.871636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 15 16:35:43.871799 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.871749    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d4bf6\""
Sep 15 16:35:43.871942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.871922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 15 16:35:43.877856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.877835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-szxbz"]
Sep 15 16:35:43.890372 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.890348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptbp\" (UniqueName: \"kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.890470 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.890383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.990798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.990765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptbp\" (UniqueName: \"kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.990798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.990800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.991142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.991127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:43.998409 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:43.998380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptbp\" (UniqueName: \"kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp\") pod \"s3-tls-init-custom-szxbz\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") " pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:44.193671 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:44.193646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:44.462211 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:44.462182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-szxbz"]
Sep 15 16:35:44.464549 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:35:44.464522    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf072f822_1ca4_4c34_bef3_c7d6e529b45f.slice/crio-96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1 WatchSource:0}: Error finding container 96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1: Status 404 returned error can't find the container with id 96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1
Sep 15 16:35:45.425274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:45.425226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerStarted","Data":"c0006d57d274b63f4b1fe8507b657631ec90dfe83ccdd21f19e5c7cf2a993050"}
Sep 15 16:35:45.425274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:45.425279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerStarted","Data":"96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1"}
Sep 15 16:35:46.430889 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:46.430863    2578 generic.go:356] "Generic (PLEG): container finished" podID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerID="c0006d57d274b63f4b1fe8507b657631ec90dfe83ccdd21f19e5c7cf2a993050" exitCode=0
Sep 15 16:35:46.431252 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:46.430944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerDied","Data":"c0006d57d274b63f4b1fe8507b657631ec90dfe83ccdd21f19e5c7cf2a993050"}
Sep 15 16:35:47.437590 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:47.437552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerStarted","Data":"2700ff985223f521b9680cc70c4ca4b4c2cc5ffcd8ac132dca1024d9c0fc2de1"}
Sep 15 16:35:47.455252 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:47.455197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-szxbz" podStartSLOduration=2.589921885 podStartE2EDuration="4.455181213s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.865259328s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:47.452982576 +0000 UTC m=+518.839621844" watchObservedRunningTime="2026-09-15 16:35:47.455181213 +0000 UTC m=+518.841820481"
Sep 15 16:35:52.460255 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:52.460221    2578 generic.go:356] "Generic (PLEG): container finished" podID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerID="2700ff985223f521b9680cc70c4ca4b4c2cc5ffcd8ac132dca1024d9c0fc2de1" exitCode=0
Sep 15 16:35:52.460631 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:52.460293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerDied","Data":"2700ff985223f521b9680cc70c4ca4b4c2cc5ffcd8ac132dca1024d9c0fc2de1"}
Sep 15 16:35:53.598571 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.598549    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:53.682535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.682504    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ptbp\" (UniqueName: \"kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp\") pod \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") "
Sep 15 16:35:53.682698 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.682595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache\") pod \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\" (UID: \"f072f822-1ca4-4c34-bef3-c7d6e529b45f\") "
Sep 15 16:35:53.684668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.684643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp" (OuterVolumeSpecName: "kube-api-access-9ptbp") pod "f072f822-1ca4-4c34-bef3-c7d6e529b45f" (UID: "f072f822-1ca4-4c34-bef3-c7d6e529b45f"). InnerVolumeSpecName "kube-api-access-9ptbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:35:53.704419 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.704378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f072f822-1ca4-4c34-bef3-c7d6e529b45f" (UID: "f072f822-1ca4-4c34-bef3-c7d6e529b45f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:35:53.783351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.783286    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptbp\" (UniqueName: \"kubernetes.io/projected/f072f822-1ca4-4c34-bef3-c7d6e529b45f-kube-api-access-9ptbp\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:35:53.783351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:53.783312    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f072f822-1ca4-4c34-bef3-c7d6e529b45f-model-cache\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:35:54.470015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:54.469986    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-szxbz"
Sep 15 16:35:54.470015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:54.469996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-szxbz" event={"ID":"f072f822-1ca4-4c34-bef3-c7d6e529b45f","Type":"ContainerDied","Data":"96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1"}
Sep 15 16:35:54.470218 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:54.470029    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bbc434a79a7374c8ab5b7c499f81d9c3d6b149b5173cb0deefe475d60186c1"
Sep 15 16:35:55.081390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.081358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"]
Sep 15 16:35:55.081881 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.081861    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerName="download-hf-model"
Sep 15 16:35:55.081953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.081884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerName="download-hf-model"
Sep 15 16:35:55.081953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.081910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerName="s3-tls-init-custom"
Sep 15 16:35:55.081953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.081919    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerName="s3-tls-init-custom"
Sep 15 16:35:55.082093 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.082024    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f072f822-1ca4-4c34-bef3-c7d6e529b45f" containerName="s3-tls-init-custom"
Sep 15 16:35:55.084588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.084568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.086970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.086947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 15 16:35:55.087092 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.087002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 15 16:35:55.087155 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.087091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d4bf6\""
Sep 15 16:35:55.091942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.091920    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"]
Sep 15 16:35:55.194855 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.194822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.194855 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.194857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cb6\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-kube-api-access-b5cb6\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.195052 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.194895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.295676 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.295638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.295834 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.295682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cb6\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-kube-api-access-b5cb6\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.295890 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.295867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.295983 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.295963    2578 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 15 16:35:55.296038 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.295986    2578 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-5h76h: secret "seaweedfs-tls-serving" not found
Sep 15 16:35:55.296038 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.296022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.296133 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.296052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving podName:12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6 nodeName:}" failed. No retries permitted until 2026-09-15 16:35:55.796030015 +0000 UTC m=+527.182669274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-5h76h" (UID: "12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6") : secret "seaweedfs-tls-serving" not found
Sep 15 16:35:55.306707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.306677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cb6\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-kube-api-access-b5cb6\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.801544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:55.801496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:55.801730 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.801670    2578 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 15 16:35:55.801730 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.801689    2578 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-5h76h: secret "seaweedfs-tls-serving" not found
Sep 15 16:35:55.801837 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:35:55.801764    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving podName:12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6 nodeName:}" failed. No retries permitted until 2026-09-15 16:35:56.801730935 +0000 UTC m=+528.188370180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-5h76h" (UID: "12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6") : secret "seaweedfs-tls-serving" not found
Sep 15 16:35:56.811214 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:56.811173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:56.813614 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:56.813593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-5h76h\" (UID: \"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:56.894722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:56.894694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"
Sep 15 16:35:57.161722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:57.161698    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-5h76h"]
Sep 15 16:35:57.486005 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:35:57.485927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h" event={"ID":"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6","Type":"ContainerStarted","Data":"d23d1e2cc56cbc089697cc7329c613969e1440992913a1eba74c02f8de228bb8"}
Sep 15 16:36:00.500521 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:00.500437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h" event={"ID":"12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6","Type":"ContainerStarted","Data":"927c519abda3f6e88e0feca9fc16b9b1a3c30b9fa4e59620d42a41548eb80aa3"}
Sep 15 16:36:00.515569 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:00.515523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-5h76h" podStartSLOduration=2.51310193 podStartE2EDuration="5.515508199s" totalImagesPullingTime="3.002406269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:00.514489179 +0000 UTC m=+531.901128448" watchObservedRunningTime="2026-09-15 16:36:00.515508199 +0000 UTC m=+531.902147467"
Sep 15 16:36:01.062901 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.062870    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-szv5s"]
Sep 15 16:36:01.065921 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.065904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.067997 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.067976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 15 16:36:01.073683 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.073660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-szv5s"]
Sep 15 16:36:01.149866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.149835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.150024 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.149938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgf4\" (UniqueName: \"kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.251279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.251244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgf4\" (UniqueName: \"kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.251447 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.251296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.251623 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.251607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.259437 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.259406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgf4\" (UniqueName: \"kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4\") pod \"s3-tls-init-serving-szv5s\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") " pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.383891 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.383864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:01.654063 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:01.653985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-szv5s"]
Sep 15 16:36:01.657375 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:36:01.657351    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dc33fd4_fb02_4858_86ec_d2dbcd386344.slice/crio-cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373 WatchSource:0}: Error finding container cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373: Status 404 returned error can't find the container with id cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373
Sep 15 16:36:02.511298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:02.511263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerStarted","Data":"0681c35ee26fd6ad1def34beec71c5f11e71a71e61e86b7ae9ae96c92a61e58a"}
Sep 15 16:36:02.511298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:02.511304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerStarted","Data":"cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373"}
Sep 15 16:36:04.521673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:04.521640    2578 generic.go:356] "Generic (PLEG): container finished" podID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerID="0681c35ee26fd6ad1def34beec71c5f11e71a71e61e86b7ae9ae96c92a61e58a" exitCode=0
Sep 15 16:36:04.522088 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:04.521717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerDied","Data":"0681c35ee26fd6ad1def34beec71c5f11e71a71e61e86b7ae9ae96c92a61e58a"}
Sep 15 16:36:05.527895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:05.527861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerStarted","Data":"3834c69c55d74a97f1effc473e152c692ae0be801f1a7c1daa10814b11e890af"}
Sep 15 16:36:05.544971 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:05.544926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-szv5s" podStartSLOduration=2.725026447 podStartE2EDuration="4.544909983s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.819883536s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:05.543272073 +0000 UTC m=+536.929911341" watchObservedRunningTime="2026-09-15 16:36:05.544909983 +0000 UTC m=+536.931549250"
Sep 15 16:36:10.555874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:10.555840    2578 generic.go:356] "Generic (PLEG): container finished" podID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerID="3834c69c55d74a97f1effc473e152c692ae0be801f1a7c1daa10814b11e890af" exitCode=0
Sep 15 16:36:10.556319 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:10.555912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerDied","Data":"3834c69c55d74a97f1effc473e152c692ae0be801f1a7c1daa10814b11e890af"}
Sep 15 16:36:11.699597 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.699572    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:11.856099 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.855990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pgf4\" (UniqueName: \"kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4\") pod \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") "
Sep 15 16:36:11.856099 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.856084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache\") pod \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\" (UID: \"7dc33fd4-fb02-4858-86ec-d2dbcd386344\") "
Sep 15 16:36:11.858432 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.858403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4" (OuterVolumeSpecName: "kube-api-access-5pgf4") pod "7dc33fd4-fb02-4858-86ec-d2dbcd386344" (UID: "7dc33fd4-fb02-4858-86ec-d2dbcd386344"). InnerVolumeSpecName "kube-api-access-5pgf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:36:11.878461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.878428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache" (OuterVolumeSpecName: "model-cache") pod "7dc33fd4-fb02-4858-86ec-d2dbcd386344" (UID: "7dc33fd4-fb02-4858-86ec-d2dbcd386344"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:36:11.957115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.957084    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgf4\" (UniqueName: \"kubernetes.io/projected/7dc33fd4-fb02-4858-86ec-d2dbcd386344-kube-api-access-5pgf4\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:11.957213 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:11.957135    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc33fd4-fb02-4858-86ec-d2dbcd386344-model-cache\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:12.566275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:12.566251    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-szv5s"
Sep 15 16:36:12.566430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:12.566242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-szv5s" event={"ID":"7dc33fd4-fb02-4858-86ec-d2dbcd386344","Type":"ContainerDied","Data":"cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373"}
Sep 15 16:36:12.566430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:12.566365    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0c6d075ccc8bc090688eb4544ef5375c1f5928503f8050187a7a4cd7d0c373"
Sep 15 16:36:24.635762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.635719    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:36:24.636250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.636075    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerName="download-hf-model"
Sep 15 16:36:24.636250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.636087    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerName="download-hf-model"
Sep 15 16:36:24.636250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.636111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerName="s3-tls-init-serving"
Sep 15 16:36:24.636250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.636117    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerName="s3-tls-init-serving"
Sep 15 16:36:24.636250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.636168    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc33fd4-fb02-4858-86ec-d2dbcd386344" containerName="s3-tls-init-serving"
Sep 15 16:36:24.639147 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.639129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.641621 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.641594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 15 16:36:24.641862 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.641846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:36:24.641958 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.641912    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:36:24.642042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.642023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:36:24.642102 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.642041    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 15 16:36:24.642102 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.642067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:36:24.642633 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.642620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:36:24.651500 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.651476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:36:24.756522 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.756493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.756669 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.756533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbfl\" (UniqueName: \"kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.756669 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.756561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.756669 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.756584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.756970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.756685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.841650 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.841615    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:24.844249 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.844233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:24.849111 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.849092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-76d9b9-kube-rbac-proxy-sar-config\""
Sep 15 16:36:24.849249 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.849148    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-76d9b9-predictor-serving-cert\""
Sep 15 16:36:24.849249 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.849176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-76d9b9-dockercfg-s9zg6\""
Sep 15 16:36:24.849436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.849312    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-76d9b9\""
Sep 15 16:36:24.857145 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.857270 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbfl\" (UniqueName: \"kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.857270 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.857270 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.857430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.857717 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.858015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.858068 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.857995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.859899 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.859879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.873325 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.873302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:24.875448 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.875426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbfl\" (UniqueName: \"kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl\") pod \"isvc-sklearn-batcher-predictor-57f7b75977-m6g5m\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.949170 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.949096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:24.958554 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.958530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:24.958653 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.958563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:24.958653 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.958582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhkr\" (UniqueName: \"kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:24.958653 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.958614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:24.958805 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:24.958753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059307 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059307 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059539 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhkr\" (UniqueName: \"kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059539 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059539 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.059858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.059839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.060037 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.060018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.060138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.060121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.061977 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.061957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.067624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.067603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhkr\" (UniqueName: \"kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr\") pod \"isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") " pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.160506 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.160472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:25.222038 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.222011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:36:25.224722 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:36:25.224696    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4cc472_229f_4bcd_a4c5_7933a8753ef5.slice/crio-1024e56177788796f14ee045962cad08e2feef12adec9713e7579d2456d86fea WatchSource:0}: Error finding container 1024e56177788796f14ee045962cad08e2feef12adec9713e7579d2456d86fea: Status 404 returned error can't find the container with id 1024e56177788796f14ee045962cad08e2feef12adec9713e7579d2456d86fea
Sep 15 16:36:25.432338 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.432305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:25.434637 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:36:25.434594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab8f7d3_f1a4_4fa0_8824_6582926f6011.slice/crio-faaea4095b3905b99a7699dd55da44fa0d346ae8f222e329f4fff98dd080200e WatchSource:0}: Error finding container faaea4095b3905b99a7699dd55da44fa0d346ae8f222e329f4fff98dd080200e: Status 404 returned error can't find the container with id faaea4095b3905b99a7699dd55da44fa0d346ae8f222e329f4fff98dd080200e
Sep 15 16:36:25.627988 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.627944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerStarted","Data":"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"}
Sep 15 16:36:25.628181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.628020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerStarted","Data":"1024e56177788796f14ee045962cad08e2feef12adec9713e7579d2456d86fea"}
Sep 15 16:36:25.630045 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.630013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv" event={"ID":"7ab8f7d3-f1a4-4fa0-8824-6582926f6011","Type":"ContainerStarted","Data":"cb5a9f2926a840b698747535e88e13f6157e535fd77bf99e89e0f261a9af6640"}
Sep 15 16:36:25.630180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:25.630085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv" event={"ID":"7ab8f7d3-f1a4-4fa0-8824-6582926f6011","Type":"ContainerStarted","Data":"faaea4095b3905b99a7699dd55da44fa0d346ae8f222e329f4fff98dd080200e"}
Sep 15 16:36:29.648173 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:29.648133    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerID="e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f" exitCode=0
Sep 15 16:36:29.648536 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:29.648181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerDied","Data":"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"}
Sep 15 16:36:32.671000 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:32.670964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv_7ab8f7d3-f1a4-4fa0-8824-6582926f6011/storage-initializer/0.log"
Sep 15 16:36:32.671461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:32.671009    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011" containerID="cb5a9f2926a840b698747535e88e13f6157e535fd77bf99e89e0f261a9af6640" exitCode=1
Sep 15 16:36:32.671461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:32.671136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv" event={"ID":"7ab8f7d3-f1a4-4fa0-8824-6582926f6011","Type":"ContainerDied","Data":"cb5a9f2926a840b698747535e88e13f6157e535fd77bf99e89e0f261a9af6640"}
Sep 15 16:36:34.441902 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:36:34.441855    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="init container storage-initializer start failed in pod isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv_kserve-ci-e2e-test(7ab8f7d3-f1a4-4fa0-8824-6582926f6011): CreateContainerConfigError: secret \"fail-s3-secret-76d9b9\" not found" logger="UnhandledError"
Sep 15 16:36:34.442385 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:36:34.441917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CreateContainerConfigError: \"secret \\\"fail-s3-secret-76d9b9\\\" not found\"" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv" podUID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011"
Sep 15 16:36:34.520159 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:34.519989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:35.039776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.039731    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv_7ab8f7d3-f1a4-4fa0-8824-6582926f6011/storage-initializer/0.log"
Sep 15 16:36:35.041176 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.041155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:35.159547 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls\") pod \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") "
Sep 15 16:36:35.159702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config\") pod \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") "
Sep 15 16:36:35.159702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzhkr\" (UniqueName: \"kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr\") pod \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") "
Sep 15 16:36:35.159702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159657    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location\") pod \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") "
Sep 15 16:36:35.159702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159676    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert\") pod \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\" (UID: \"7ab8f7d3-f1a4-4fa0-8824-6582926f6011\") "
Sep 15 16:36:35.159991 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.159939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ab8f7d3-f1a4-4fa0-8824-6582926f6011" (UID: "7ab8f7d3-f1a4-4fa0-8824-6582926f6011"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:36:35.160051 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.160018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ab8f7d3-f1a4-4fa0-8824-6582926f6011" (UID: "7ab8f7d3-f1a4-4fa0-8824-6582926f6011"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:36:35.160092 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.160062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ab8f7d3-f1a4-4fa0-8824-6582926f6011" (UID: "7ab8f7d3-f1a4-4fa0-8824-6582926f6011"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:36:35.161893 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.161868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ab8f7d3-f1a4-4fa0-8824-6582926f6011" (UID: "7ab8f7d3-f1a4-4fa0-8824-6582926f6011"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:36:35.162227 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.162209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr" (OuterVolumeSpecName: "kube-api-access-pzhkr") pod "7ab8f7d3-f1a4-4fa0-8824-6582926f6011" (UID: "7ab8f7d3-f1a4-4fa0-8824-6582926f6011"). InnerVolumeSpecName "kube-api-access-pzhkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:36:35.261074 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.261045    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:35.261272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.261193    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhkr\" (UniqueName: \"kubernetes.io/projected/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kube-api-access-pzhkr\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:35.261272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.261221    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:35.261272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.261238    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:35.261272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.261254    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ab8f7d3-f1a4-4fa0-8824-6582926f6011-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:36:35.692098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.692074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv_7ab8f7d3-f1a4-4fa0-8824-6582926f6011/storage-initializer/0.log"
Sep 15 16:36:35.692544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.692195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"
Sep 15 16:36:35.692544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.692212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv" event={"ID":"7ab8f7d3-f1a4-4fa0-8824-6582926f6011","Type":"ContainerDied","Data":"faaea4095b3905b99a7699dd55da44fa0d346ae8f222e329f4fff98dd080200e"}
Sep 15 16:36:35.692544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.692291    2578 scope.go:117] "RemoveContainer" containerID="cb5a9f2926a840b698747535e88e13f6157e535fd77bf99e89e0f261a9af6640"
Sep 15 16:36:35.729169 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.729142    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:35.735147 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:35.735115    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-76d9b9-predictor-6d9dffdbd7-mj9nv"]
Sep 15 16:36:37.155375 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:37.155346    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011" path="/var/lib/kubelet/pods/7ab8f7d3-f1a4-4fa0-8824-6582926f6011/volumes"
Sep 15 16:36:39.774802 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.774766    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:36:39.775255 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.775221    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011" containerName="storage-initializer"
Sep 15 16:36:39.775255 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.775235    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011" containerName="storage-initializer"
Sep 15 16:36:39.775355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.775296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8f7d3-f1a4-4fa0-8824-6582926f6011" containerName="storage-initializer"
Sep 15 16:36:39.787221 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.787188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:39.790351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.790262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 15 16:36:39.790604 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.790577    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 15 16:36:39.797511 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.797486    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:36:39.902434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.902334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:39.902434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.902384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:39.902434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.902408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:39.902673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.902497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dl9\" (UniqueName: \"kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:39.902673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:39.902538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.003874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.003831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004057 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.003891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004057 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004184 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dl9\" (UniqueName: \"kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004244 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:36:40.004186    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 15 16:36:40.004244 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004327 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004327 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:36:40.004274    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls podName:4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6 nodeName:}" failed. No retries permitted until 2026-09-15 16:36:40.504252696 +0000 UTC m=+571.890891944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-rkblc" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 15 16:36:40.004677 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.004860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.004841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.013050 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.013013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dl9\" (UniqueName: \"kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.509970 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.509932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.512326 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.512303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-rkblc\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:40.701810 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:40.701772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:36:43.916516 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:43.916492    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:36:43.919221 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:36:43.919191    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa2bff6_3e3d_41fb_b1fe_ad274b8fd3a6.slice/crio-29b5d5e5292f8d98cca235848affcd8f836678800e2d4022a78fdd678e458f5e WatchSource:0}: Error finding container 29b5d5e5292f8d98cca235848affcd8f836678800e2d4022a78fdd678e458f5e: Status 404 returned error can't find the container with id 29b5d5e5292f8d98cca235848affcd8f836678800e2d4022a78fdd678e458f5e
Sep 15 16:36:44.745343 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:44.745304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerStarted","Data":"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"}
Sep 15 16:36:44.745546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:44.745376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerStarted","Data":"29b5d5e5292f8d98cca235848affcd8f836678800e2d4022a78fdd678e458f5e"}
Sep 15 16:36:44.747953 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:44.747921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerStarted","Data":"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"}
Sep 15 16:36:46.760225 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:46.760189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerStarted","Data":"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"}
Sep 15 16:36:48.773673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:48.773594    2578 generic.go:356] "Generic (PLEG): container finished" podID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerID="961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33" exitCode=0
Sep 15 16:36:48.773673 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:48.773653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerDied","Data":"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"}
Sep 15 16:36:50.792305 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.792261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerStarted","Data":"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"}
Sep 15 16:36:50.795146 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.795119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:50.795275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.795194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:50.795275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.795242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:50.795642 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.795607    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:36:50.796770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.796680    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:36:50.818995 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:50.818859    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podStartSLOduration=3.032786826 podStartE2EDuration="26.818838872s" totalImagesPullingTime="20.06181121s" totalInitContainerRuntime="3.724240836s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:50.816314639 +0000 UTC m=+582.202953907" watchObservedRunningTime="2026-09-15 16:36:50.818838872 +0000 UTC m=+582.205478142"
Sep 15 16:36:51.804689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:51.804407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:36:51.805257 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:51.805148    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:36:51.809865 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:51.809840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:36:52.811591 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:52.811224    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:36:52.812084 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:52.811880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:36:53.818894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:53.818848    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:36:53.819439 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:36:53.819375    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:03.818971 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:03.818930    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:03.819474 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:03.819236    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:12.927669 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:12.927629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerStarted","Data":"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"}
Sep 15 16:37:12.928113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:12.927691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerStarted","Data":"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"}
Sep 15 16:37:12.928113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:12.927888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:37:12.950118 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:12.950070    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podStartSLOduration=6.580158153 podStartE2EDuration="33.950055141s" totalImagesPullingTime="23.195586288s" totalInitContainerRuntime="4.1743107s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:37:12.947870685 +0000 UTC m=+604.334509953" watchObservedRunningTime="2026-09-15 16:37:12.950055141 +0000 UTC m=+604.336694408"
Sep 15 16:37:13.818892 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:13.818842    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:13.819087 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:13.819033    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:13.932265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:13.932234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:37:13.933623 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:13.933598    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:14.937137 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:14.937100    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:19.941404 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:19.941375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:37:19.941992 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:19.941961    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:23.818998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:23.818951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:23.819404 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:23.819300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:29.942788 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:29.942728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:33.818886 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:33.818798    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:33.819230 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:33.818939    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:39.942498 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:39.942456    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:43.818950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:43.818910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:43.819303 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:43.819056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:49.942110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:49.942062    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:37:53.819076 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:53.819022    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:37:53.820682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:53.819530    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:37:59.942980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:37:59.942944    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:38:03.818947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:03.818908    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:03.819576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:03.819557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:09.592181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.592143    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:38:09.592603 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.592573    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" containerID="cri-o://3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f" gracePeriod=30
Sep 15 16:38:09.592679 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.592591    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" containerID="cri-o://30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289" gracePeriod=30
Sep 15 16:38:09.592810 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.592769    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" containerID="cri-o://90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de" gracePeriod=30
Sep 15 16:38:09.710753 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.710708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:38:09.713892 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.713870    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.716160 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.716125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 15 16:38:09.716287 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.716183    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 15 16:38:09.728414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.728392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:38:09.822609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.822572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.822609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.822612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.822847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.822653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.822847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.822680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.822847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.822801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92dgv\" (UniqueName: \"kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92dgv\" (UniqueName: \"kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924565 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.924924 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.925012 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.924952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.926765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.926732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.932888 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.932855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92dgv\" (UniqueName: \"kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv\") pod \"isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:09.942832 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:09.942806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:38:10.024239 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:10.024203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:10.210678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:10.210600    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerID="90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de" exitCode=2
Sep 15 16:38:10.210678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:10.210648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerDied","Data":"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"}
Sep 15 16:38:10.303978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:10.301559    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:38:10.305127 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:38:10.305097    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67feb104_406a_45ea_95c2_f590f94c3998.slice/crio-d7ad1695f7caeee0d659d3279a678691a118cd9580471edc06a65286b2dc3aca WatchSource:0}: Error finding container d7ad1695f7caeee0d659d3279a678691a118cd9580471edc06a65286b2dc3aca: Status 404 returned error can't find the container with id d7ad1695f7caeee0d659d3279a678691a118cd9580471edc06a65286b2dc3aca
Sep 15 16:38:11.216619 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:11.216582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerStarted","Data":"44070b0c4f5966a1ccb37886a28405f34381fd612394e4edef22a8f0e84c7c73"}
Sep 15 16:38:11.217041 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:11.216652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerStarted","Data":"d7ad1695f7caeee0d659d3279a678691a118cd9580471edc06a65286b2dc3aca"}
Sep 15 16:38:11.804947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:11.804903    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:13.818950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:13.818908    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:13.820080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:13.820053    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:38:14.233711 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:14.233675    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerID="3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f" exitCode=0
Sep 15 16:38:14.233866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:14.233761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerDied","Data":"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"}
Sep 15 16:38:15.239639 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:15.239606    2578 generic.go:356] "Generic (PLEG): container finished" podID="67feb104-406a-45ea-95c2-f590f94c3998" containerID="44070b0c4f5966a1ccb37886a28405f34381fd612394e4edef22a8f0e84c7c73" exitCode=0
Sep 15 16:38:15.240074 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:15.239667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerDied","Data":"44070b0c4f5966a1ccb37886a28405f34381fd612394e4edef22a8f0e84c7c73"}
Sep 15 16:38:16.246696 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.246664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerStarted","Data":"98d7eee15bdf1723f81cb613346aee50c59973d35d948ca81e2c1dcdca3b0acf"}
Sep 15 16:38:16.247122 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.246730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerStarted","Data":"5184e60ee0725183475ae82beb9fa33c57b17245ed6cfcaf33317a31b99885c0"}
Sep 15 16:38:16.247122 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.246793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerStarted","Data":"6379114e52ac06a2a3623163d97e00530f1bc1870b7226bff991e6e7b8555222"}
Sep 15 16:38:16.247122 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.247041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:16.247262 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.247183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:16.248498 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.248465    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:16.804658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:16.804621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:17.252647 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:17.252616    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:17.253122 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:17.252789    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:17.253874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:17.253841    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:18.257232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:18.257194    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:18.257747 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:18.257705    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:19.942211 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:19.942169    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:38:21.804415 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:21.804376    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:21.804886 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:21.804531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:21.825068 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:21.825015    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podStartSLOduration=8.859881625 podStartE2EDuration="12.824999954s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965118329s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:38:16.267599921 +0000 UTC m=+667.654239212" watchObservedRunningTime="2026-09-15 16:38:21.824999954 +0000 UTC m=+673.211639215"
Sep 15 16:38:23.261134 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:23.261104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:38:23.261749 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:23.261706    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:23.262030 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:23.262002    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:23.819157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:23.819113    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:23.819847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:23.819818    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:38:26.804180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:26.804137    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:29.942896 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:29.942863    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:38:31.805048 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:31.805007    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:33.262404 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.262360    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:33.262867 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.262809    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:33.818661 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.818622    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:33.818858 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.818770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:33.819874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.819852    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:38:33.819978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:33.819964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:36.804750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:36.804699    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:38:39.737944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.737921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:39.780782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.780719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:38:39.781370 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.781337    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" containerID="cri-o://2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06" gracePeriod=30
Sep 15 16:38:39.786660 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.781775    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy" containerID="cri-o://72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a" gracePeriod=30
Sep 15 16:38:39.797844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.797820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location\") pod \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") "
Sep 15 16:38:39.797940 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.797885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls\") pod \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") "
Sep 15 16:38:39.797989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.797959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert\") pod \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") "
Sep 15 16:38:39.798040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.797995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzbfl\" (UniqueName: \"kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl\") pod \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") "
Sep 15 16:38:39.798040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config\") pod \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\" (UID: \"2e4cc472-229f-4bcd-a4c5-7933a8753ef5\") "
Sep 15 16:38:39.798141 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e4cc472-229f-4bcd-a4c5-7933a8753ef5" (UID: "2e4cc472-229f-4bcd-a4c5-7933a8753ef5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:38:39.798356 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e4cc472-229f-4bcd-a4c5-7933a8753ef5" (UID: "2e4cc472-229f-4bcd-a4c5-7933a8753ef5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:38:39.798435 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798420    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:39.798491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798440    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:39.798491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.798436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e4cc472-229f-4bcd-a4c5-7933a8753ef5" (UID: "2e4cc472-229f-4bcd-a4c5-7933a8753ef5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:38:39.800036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.800013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e4cc472-229f-4bcd-a4c5-7933a8753ef5" (UID: "2e4cc472-229f-4bcd-a4c5-7933a8753ef5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:38:39.800390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.800365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl" (OuterVolumeSpecName: "kube-api-access-mzbfl") pod "2e4cc472-229f-4bcd-a4c5-7933a8753ef5" (UID: "2e4cc472-229f-4bcd-a4c5-7933a8753ef5"). InnerVolumeSpecName "kube-api-access-mzbfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:38:39.899180 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.899158    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:39.899269 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.899181    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbfl\" (UniqueName: \"kubernetes.io/projected/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-api-access-mzbfl\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:39.899269 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.899191    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e4cc472-229f-4bcd-a4c5-7933a8753ef5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:39.937989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.937955    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 15 16:38:39.942304 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:39.942278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 15 16:38:40.367241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.367148    2578 generic.go:356] "Generic (PLEG): container finished" podID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerID="72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a" exitCode=2
Sep 15 16:38:40.367241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.367220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerDied","Data":"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"}
Sep 15 16:38:40.369394 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.369371    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerID="30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289" exitCode=0
Sep 15 16:38:40.369552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.369440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerDied","Data":"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"}
Sep 15 16:38:40.369552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.369461    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"
Sep 15 16:38:40.369552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.369505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m" event={"ID":"2e4cc472-229f-4bcd-a4c5-7933a8753ef5","Type":"ContainerDied","Data":"1024e56177788796f14ee045962cad08e2feef12adec9713e7579d2456d86fea"}
Sep 15 16:38:40.369552 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.369548    2578 scope.go:117] "RemoveContainer" containerID="30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"
Sep 15 16:38:40.378386 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.378372    2578 scope.go:117] "RemoveContainer" containerID="90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"
Sep 15 16:38:40.385871 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.385855    2578 scope.go:117] "RemoveContainer" containerID="3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"
Sep 15 16:38:40.392668 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.392622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:38:40.395244 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.395222    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-57f7b75977-m6g5m"]
Sep 15 16:38:40.395407 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.395391    2578 scope.go:117] "RemoveContainer" containerID="e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"
Sep 15 16:38:40.402907 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.402884    2578 scope.go:117] "RemoveContainer" containerID="30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"
Sep 15 16:38:40.403178 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:40.403159    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289\": container with ID starting with 30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289 not found: ID does not exist" containerID="30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"
Sep 15 16:38:40.403243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403191    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289"} err="failed to get container status \"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289\": rpc error: code = NotFound desc = could not find container \"30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289\": container with ID starting with 30d4ee4c9739a707695851b320bb963371769529ea3baeb0b63dc0e8f5ae4289 not found: ID does not exist"
Sep 15 16:38:40.403243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403213    2578 scope.go:117] "RemoveContainer" containerID="90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"
Sep 15 16:38:40.403478 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:40.403459    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de\": container with ID starting with 90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de not found: ID does not exist" containerID="90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"
Sep 15 16:38:40.403518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403486    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de"} err="failed to get container status \"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de\": rpc error: code = NotFound desc = could not find container \"90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de\": container with ID starting with 90a99362759521e599b4cb09cc0fb93159ae56cbcb5c70ae10d6577b22f302de not found: ID does not exist"
Sep 15 16:38:40.403518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403503    2578 scope.go:117] "RemoveContainer" containerID="3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"
Sep 15 16:38:40.403772 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:40.403752    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f\": container with ID starting with 3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f not found: ID does not exist" containerID="3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"
Sep 15 16:38:40.403821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403776    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f"} err="failed to get container status \"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f\": rpc error: code = NotFound desc = could not find container \"3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f\": container with ID starting with 3e83fff9f0dc8d1a73b964863ff45d465c8c71fd32cd798fd1a1b1825e34bb9f not found: ID does not exist"
Sep 15 16:38:40.403821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.403789    2578 scope.go:117] "RemoveContainer" containerID="e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"
Sep 15 16:38:40.404029 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:40.404011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f\": container with ID starting with e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f not found: ID does not exist" containerID="e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"
Sep 15 16:38:40.404082 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:40.404036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f"} err="failed to get container status \"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f\": rpc error: code = NotFound desc = could not find container \"e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f\": container with ID starting with e84da14a895f844b798b6a68877e7d0c8e130ad258d5c73273b2e901be1fce6f not found: ID does not exist"
Sep 15 16:38:41.154804 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:41.154773    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" path="/var/lib/kubelet/pods/2e4cc472-229f-4bcd-a4c5-7933a8753ef5/volumes"
Sep 15 16:38:43.261963 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:43.261919    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:43.262423 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:43.262400    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:44.941351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.941328    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:38:44.953576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953550    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:38:44.953927 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953914    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container"
Sep 15 16:38:44.953976 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953929    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container"
Sep 15 16:38:44.953976 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953940    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="storage-initializer"
Sep 15 16:38:44.953976 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953945    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="storage-initializer"
Sep 15 16:38:44.953976 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953953    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container"
Sep 15 16:38:44.953976 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953959    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="storage-initializer"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="storage-initializer"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.953999    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954005    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954014    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954019    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954029    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954034    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954079    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954088    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kserve-container"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954103    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="kserve-container"
Sep 15 16:38:44.954124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.954108    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e4cc472-229f-4bcd-a4c5-7933a8753ef5" containerName="agent"
Sep 15 16:38:44.956449 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.956435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:44.959141 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.959105    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 15 16:38:44.959141 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:44.959115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 15 16:38:45.008089 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.008053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:38:45.044545 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") pod \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") "
Sep 15 16:38:45.044545 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044501    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config\") pod \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") "
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2dl9\" (UniqueName: \"kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9\") pod \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") "
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert\") pod \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") "
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location\") pod \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\" (UID: \"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6\") "
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.044977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045034    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.045154 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlns9\" (UniqueName: \"kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.045540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045177    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:45.045540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045195    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:45.045540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.045214    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:45.046929 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.046909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9" (OuterVolumeSpecName: "kube-api-access-s2dl9") pod "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6"). InnerVolumeSpecName "kube-api-access-s2dl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:38:45.046929 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.046921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" (UID: "4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:38:45.146517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlns9\" (UniqueName: \"kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146673    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:45.146699 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.146689    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2dl9\" (UniqueName: \"kubernetes.io/projected/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6-kube-api-access-s2dl9\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:38:45.147070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.147051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.147388 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.147365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.147472 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.147396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.149118 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.149091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.159873 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.159848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlns9\" (UniqueName: \"kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9\") pod \"isvc-predictive-xgboost-predictor-695f978978-bbdxr\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.266789 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.266730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:45.395103 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.395073    2578 generic.go:356] "Generic (PLEG): container finished" podID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerID="2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06" exitCode=0
Sep 15 16:38:45.395267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.395123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerDied","Data":"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"}
Sep 15 16:38:45.395267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.395162    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"
Sep 15 16:38:45.395267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.395180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" event={"ID":"4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6","Type":"ContainerDied","Data":"29b5d5e5292f8d98cca235848affcd8f836678800e2d4022a78fdd678e458f5e"}
Sep 15 16:38:45.395267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.395208    2578 scope.go:117] "RemoveContainer" containerID="72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"
Sep 15 16:38:45.403413 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.403391    2578 scope.go:117] "RemoveContainer" containerID="2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"
Sep 15 16:38:45.412785 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.412759    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:38:45.414556 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.414535    2578 scope.go:117] "RemoveContainer" containerID="961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"
Sep 15 16:38:45.415466 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.415446    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc"]
Sep 15 16:38:45.422848 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.422828    2578 scope.go:117] "RemoveContainer" containerID="72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"
Sep 15 16:38:45.423119 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:45.423100    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a\": container with ID starting with 72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a not found: ID does not exist" containerID="72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"
Sep 15 16:38:45.423207 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.423129    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a"} err="failed to get container status \"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a\": rpc error: code = NotFound desc = could not find container \"72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a\": container with ID starting with 72648173b7e3b7006ab102d3bf55a532bac773aea3f2e586ac5ba02c2fbb1e5a not found: ID does not exist"
Sep 15 16:38:45.423207 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.423147    2578 scope.go:117] "RemoveContainer" containerID="2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"
Sep 15 16:38:45.423432 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:45.423412    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06\": container with ID starting with 2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06 not found: ID does not exist" containerID="2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"
Sep 15 16:38:45.423473 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.423439    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06"} err="failed to get container status \"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06\": rpc error: code = NotFound desc = could not find container \"2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06\": container with ID starting with 2485bd20926d95836a84f2a75551f19873342c8697324fa25a636c7d505ece06 not found: ID does not exist"
Sep 15 16:38:45.423473 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.423455    2578 scope.go:117] "RemoveContainer" containerID="961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"
Sep 15 16:38:45.423705 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:38:45.423687    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33\": container with ID starting with 961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33 not found: ID does not exist" containerID="961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"
Sep 15 16:38:45.423793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.423711    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33"} err="failed to get container status \"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33\": rpc error: code = NotFound desc = could not find container \"961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33\": container with ID starting with 961d61ad81a6d0be254d0ac0d6bbe82cbfb7eee1e2abb514fc8db3bd7c603a33 not found: ID does not exist"
Sep 15 16:38:45.539360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.539328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:38:45.541838 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:38:45.541809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd29f881_dc3a_4372_a71f_74c6d0635920.slice/crio-c7b9f4a7a9a3815e3950936b54584341ad7077cf024a04fba5cb8a723096df48 WatchSource:0}: Error finding container c7b9f4a7a9a3815e3950936b54584341ad7077cf024a04fba5cb8a723096df48: Status 404 returned error can't find the container with id c7b9f4a7a9a3815e3950936b54584341ad7077cf024a04fba5cb8a723096df48
Sep 15 16:38:45.938608 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:45.938560    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-rkblc" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 16:38:46.402390 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:46.402356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerStarted","Data":"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"}
Sep 15 16:38:46.402767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:46.402424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerStarted","Data":"c7b9f4a7a9a3815e3950936b54584341ad7077cf024a04fba5cb8a723096df48"}
Sep 15 16:38:47.155071 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:47.155038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6" path="/var/lib/kubelet/pods/4aa2bff6-3e3d-41fb-b1fe-ad274b8fd3a6/volumes"
Sep 15 16:38:49.417596 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:49.417568    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerID="f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52" exitCode=0
Sep 15 16:38:49.417939 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:49.417639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerDied","Data":"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"}
Sep 15 16:38:50.424120 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:50.424088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerStarted","Data":"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"}
Sep 15 16:38:50.424500 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:50.424152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerStarted","Data":"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"}
Sep 15 16:38:50.424500 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:50.424355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:50.442003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:50.441960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podStartSLOduration=2.634029667 podStartE2EDuration="6.441944898s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807915231s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:38:50.441865919 +0000 UTC m=+701.828505189" watchObservedRunningTime="2026-09-15 16:38:50.441944898 +0000 UTC m=+701.828584357"
Sep 15 16:38:51.429167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:51.429133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:51.430277 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:51.430252    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:38:52.434370 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:52.434323    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:38:53.262308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:53.262267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:38:53.262769 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:53.262730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:38:57.438506 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:57.438476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:38:57.438959 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:38:57.438933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:03.262495 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:03.262400    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:39:03.262962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:03.262856    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:07.439226 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:07.439186    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:13.262473 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:13.262421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:39:13.263016 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:13.262953    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:17.439770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:17.439706    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:23.261924 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:23.261885    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:39:23.262374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:23.262352    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:27.439949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:27.439909    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:33.261955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:33.261917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:39:33.262421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:33.262182    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:39:37.439324 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:37.439284    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:44.762658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.762615    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:39:44.763259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.762962    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" containerID="cri-o://6379114e52ac06a2a3623163d97e00530f1bc1870b7226bff991e6e7b8555222" gracePeriod=30
Sep 15 16:39:44.763259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.762979    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" containerID="cri-o://5184e60ee0725183475ae82beb9fa33c57b17245ed6cfcaf33317a31b99885c0" gracePeriod=30
Sep 15 16:39:44.763259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.763026    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" containerID="cri-o://98d7eee15bdf1723f81cb613346aee50c59973d35d948ca81e2c1dcdca3b0acf" gracePeriod=30
Sep 15 16:39:44.863603 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.863572    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:39:44.866237 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.866220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:44.868607 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.868583    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 15 16:39:44.868764 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.868631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 15 16:39:44.880413 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.880391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:39:44.961602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.961573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:44.961602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.961608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvk9g\" (UniqueName: \"kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:44.961838 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:44.961654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.062807 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.062695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvk9g\" (UniqueName: \"kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.062980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.062815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.062980 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.062900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.063471 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.063445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.065396 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.065377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.071850 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.071825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvk9g\" (UniqueName: \"kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g\") pod \"message-dumper-predictor-6575bcfdfd-4d6ld\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.177490 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.177458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:45.450722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.450697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:39:45.452774 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:39:45.452727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5163354_4370_4769_a1cb_174ec4b58675.slice/crio-64e178a06cce01ca14b37b239b393ebebd1a7eddf35c30ce7a51a3cf6fdd1b99 WatchSource:0}: Error finding container 64e178a06cce01ca14b37b239b393ebebd1a7eddf35c30ce7a51a3cf6fdd1b99: Status 404 returned error can't find the container with id 64e178a06cce01ca14b37b239b393ebebd1a7eddf35c30ce7a51a3cf6fdd1b99
Sep 15 16:39:45.454605 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.454588    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:39:45.692833 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.692788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerStarted","Data":"64e178a06cce01ca14b37b239b393ebebd1a7eddf35c30ce7a51a3cf6fdd1b99"}
Sep 15 16:39:45.695535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.695510    2578 generic.go:356] "Generic (PLEG): container finished" podID="67feb104-406a-45ea-95c2-f590f94c3998" containerID="5184e60ee0725183475ae82beb9fa33c57b17245ed6cfcaf33317a31b99885c0" exitCode=2
Sep 15 16:39:45.695657 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:45.695540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerDied","Data":"5184e60ee0725183475ae82beb9fa33c57b17245ed6cfcaf33317a31b99885c0"}
Sep 15 16:39:47.439587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.439542    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:47.710629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.710532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerStarted","Data":"aae6b5ef3e07ba3fa2803e53a7846b501bb04a10c40f9e3bb62ff7762b477d6a"}
Sep 15 16:39:47.710629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.710598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerStarted","Data":"3db266e2d5d485386bd063e2386b06ab67d350a5b74de75ade7cfdbf2050e541"}
Sep 15 16:39:47.710887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.710633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:47.710887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.710692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:47.712436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.712418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:47.727897 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:47.727853    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" podStartSLOduration=2.391646915 podStartE2EDuration="3.727839078s" totalImagesPullingTime="1.336192163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:39:47.726402216 +0000 UTC m=+759.113041484" watchObservedRunningTime="2026-09-15 16:39:47.727839078 +0000 UTC m=+759.114478342"
Sep 15 16:39:48.258355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:48.258316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:39:50.728645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:50.728614    2578 generic.go:356] "Generic (PLEG): container finished" podID="67feb104-406a-45ea-95c2-f590f94c3998" containerID="6379114e52ac06a2a3623163d97e00530f1bc1870b7226bff991e6e7b8555222" exitCode=0
Sep 15 16:39:50.729097 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:50.728672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerDied","Data":"6379114e52ac06a2a3623163d97e00530f1bc1870b7226bff991e6e7b8555222"}
Sep 15 16:39:53.258137 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:53.258093    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:39:53.262613 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:53.262578    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:39:53.262971 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:53.262942    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:54.725038 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:54.725008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:39:57.439206 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:57.439169    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:39:58.258036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:58.257990    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:39:58.258207 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:39:58.258137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:40:03.257716 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:03.257671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:40:03.261973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:03.261943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:40:03.262304 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:03.262280    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:04.900910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:04.900867    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:40:04.904290 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:04.904270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:04.906542 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:04.906520    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 15 16:40:04.906656 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:04.906637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 15 16:40:04.914414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:04.914390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:40:05.034127 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.034093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn89\" (UniqueName: \"kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.034294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.034156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.034294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.034225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.034294 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.034278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.034399 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.034307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135185 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn89\" (UniqueName: \"kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135372 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135372 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135372 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.135793 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.135772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.136070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.136048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.136138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.136118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.138123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.138103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.145715 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.145685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn89\" (UniqueName: \"kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89\") pod \"isvc-logger-predictor-56c548f55d-p6rbc\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.214655 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.214575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:05.491287 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.491259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:40:05.493823 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:40:05.493795    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e1bb7b_e78e_4601_9614_a756e42d39db.slice/crio-99782e8a42fd665923871fa9941c13c833e5ee658ef1ab64c041d596e289628e WatchSource:0}: Error finding container 99782e8a42fd665923871fa9941c13c833e5ee658ef1ab64c041d596e289628e: Status 404 returned error can't find the container with id 99782e8a42fd665923871fa9941c13c833e5ee658ef1ab64c041d596e289628e
Sep 15 16:40:05.803415 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.803333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerStarted","Data":"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"}
Sep 15 16:40:05.803415 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:05.803411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerStarted","Data":"99782e8a42fd665923871fa9941c13c833e5ee658ef1ab64c041d596e289628e"}
Sep 15 16:40:07.439464 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:07.439436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:40:08.257757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:08.257702    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:40:09.824860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:09.824773    2578 generic.go:356] "Generic (PLEG): container finished" podID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerID="bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe" exitCode=0
Sep 15 16:40:09.824860 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:09.824823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerDied","Data":"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"}
Sep 15 16:40:10.831705 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:10.831670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerStarted","Data":"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"}
Sep 15 16:40:10.832201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:10.831771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerStarted","Data":"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"}
Sep 15 16:40:10.832201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:10.831814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerStarted","Data":"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"}
Sep 15 16:40:10.832201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:10.832053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:10.854050 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:10.853993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podStartSLOduration=2.868910703 podStartE2EDuration="6.853971357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.985060654s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:10.852274705 +0000 UTC m=+782.238913975" watchObservedRunningTime="2026-09-15 16:40:10.853971357 +0000 UTC m=+782.240610626"
Sep 15 16:40:11.836797 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:11.836766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:11.837184 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:11.836842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:11.838395 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:11.838364    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:11.839031 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:11.839004    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:12.841963 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:12.841919    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:12.842395 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:12.842221    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:13.257790 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:13.257723    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:40:13.262119 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:13.262094    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Sep 15 16:40:13.262224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:13.262211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:40:13.262526 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:13.262504    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:13.262603 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:13.262594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:40:14.855055 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:14.855012    2578 generic.go:356] "Generic (PLEG): container finished" podID="67feb104-406a-45ea-95c2-f590f94c3998" containerID="98d7eee15bdf1723f81cb613346aee50c59973d35d948ca81e2c1dcdca3b0acf" exitCode=0
Sep 15 16:40:14.855434 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:14.855078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerDied","Data":"98d7eee15bdf1723f81cb613346aee50c59973d35d948ca81e2c1dcdca3b0acf"}
Sep 15 16:40:14.906123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:14.906100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:40:15.030124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls\") pod \"67feb104-406a-45ea-95c2-f590f94c3998\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") "
Sep 15 16:40:15.030124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030098    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert\") pod \"67feb104-406a-45ea-95c2-f590f94c3998\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") "
Sep 15 16:40:15.030412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config\") pod \"67feb104-406a-45ea-95c2-f590f94c3998\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") "
Sep 15 16:40:15.030412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location\") pod \"67feb104-406a-45ea-95c2-f590f94c3998\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") "
Sep 15 16:40:15.030412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030285    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92dgv\" (UniqueName: \"kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv\") pod \"67feb104-406a-45ea-95c2-f590f94c3998\" (UID: \"67feb104-406a-45ea-95c2-f590f94c3998\") "
Sep 15 16:40:15.030562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030511    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67feb104-406a-45ea-95c2-f590f94c3998" (UID: "67feb104-406a-45ea-95c2-f590f94c3998"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:15.030562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67feb104-406a-45ea-95c2-f590f94c3998" (UID: "67feb104-406a-45ea-95c2-f590f94c3998"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:15.030698 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.030670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67feb104-406a-45ea-95c2-f590f94c3998" (UID: "67feb104-406a-45ea-95c2-f590f94c3998"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:40:15.031535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.031512    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:40:15.031805 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.031781    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" containerID="cri-o://fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55" gracePeriod=30
Sep 15 16:40:15.032107 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.031974    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kube-rbac-proxy" containerID="cri-o://3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918" gracePeriod=30
Sep 15 16:40:15.032574 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.032488    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67feb104-406a-45ea-95c2-f590f94c3998" (UID: "67feb104-406a-45ea-95c2-f590f94c3998"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:15.032979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.032945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv" (OuterVolumeSpecName: "kube-api-access-92dgv") pod "67feb104-406a-45ea-95c2-f590f94c3998" (UID: "67feb104-406a-45ea-95c2-f590f94c3998"). InnerVolumeSpecName "kube-api-access-92dgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:15.131355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.131324    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67feb104-406a-45ea-95c2-f590f94c3998-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:15.131355 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.131353    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:15.131546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.131365    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67feb104-406a-45ea-95c2-f590f94c3998-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:15.131546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.131378    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67feb104-406a-45ea-95c2-f590f94c3998-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:15.131546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.131390    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92dgv\" (UniqueName: \"kubernetes.io/projected/67feb104-406a-45ea-95c2-f590f94c3998-kube-api-access-92dgv\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:15.862388 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.862355    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerID="3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918" exitCode=2
Sep 15 16:40:15.862827 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.862434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerDied","Data":"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"}
Sep 15 16:40:15.864686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.864665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw" event={"ID":"67feb104-406a-45ea-95c2-f590f94c3998","Type":"ContainerDied","Data":"d7ad1695f7caeee0d659d3279a678691a118cd9580471edc06a65286b2dc3aca"}
Sep 15 16:40:15.864806 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.864701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"
Sep 15 16:40:15.864806 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.864725    2578 scope.go:117] "RemoveContainer" containerID="98d7eee15bdf1723f81cb613346aee50c59973d35d948ca81e2c1dcdca3b0acf"
Sep 15 16:40:15.873070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.873056    2578 scope.go:117] "RemoveContainer" containerID="5184e60ee0725183475ae82beb9fa33c57b17245ed6cfcaf33317a31b99885c0"
Sep 15 16:40:15.880814 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.880794    2578 scope.go:117] "RemoveContainer" containerID="6379114e52ac06a2a3623163d97e00530f1bc1870b7226bff991e6e7b8555222"
Sep 15 16:40:15.883907 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.883886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:40:15.888203 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.888177    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-559d5db67b-6glrw"]
Sep 15 16:40:15.891654 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:15.891635    2578 scope.go:117] "RemoveContainer" containerID="44070b0c4f5966a1ccb37886a28405f34381fd612394e4edef22a8f0e84c7c73"
Sep 15 16:40:17.154499 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.154462    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67feb104-406a-45ea-95c2-f590f94c3998" path="/var/lib/kubelet/pods/67feb104-406a-45ea-95c2-f590f94c3998/volumes"
Sep 15 16:40:17.435112 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.435020    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 15 16:40:17.439323 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.439299    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:40:17.845867 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.845789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:40:17.846458 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.846410    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:17.846638 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:17.846618    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:19.773000 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.772977    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:40:19.864774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.864669    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location\") pod \"dd29f881-dc3a-4372-a71f-74c6d0635920\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") "
Sep 15 16:40:19.864774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.864720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert\") pod \"dd29f881-dc3a-4372-a71f-74c6d0635920\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") "
Sep 15 16:40:19.864774 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.864762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlns9\" (UniqueName: \"kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9\") pod \"dd29f881-dc3a-4372-a71f-74c6d0635920\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") "
Sep 15 16:40:19.865056 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.864791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls\") pod \"dd29f881-dc3a-4372-a71f-74c6d0635920\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") "
Sep 15 16:40:19.865056 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.864824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config\") pod \"dd29f881-dc3a-4372-a71f-74c6d0635920\" (UID: \"dd29f881-dc3a-4372-a71f-74c6d0635920\") "
Sep 15 16:40:19.865165 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.865105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd29f881-dc3a-4372-a71f-74c6d0635920" (UID: "dd29f881-dc3a-4372-a71f-74c6d0635920"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:40:19.865219 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.865193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd29f881-dc3a-4372-a71f-74c6d0635920" (UID: "dd29f881-dc3a-4372-a71f-74c6d0635920"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:19.865263 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.865236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd29f881-dc3a-4372-a71f-74c6d0635920" (UID: "dd29f881-dc3a-4372-a71f-74c6d0635920"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:19.867111 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.867080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd29f881-dc3a-4372-a71f-74c6d0635920" (UID: "dd29f881-dc3a-4372-a71f-74c6d0635920"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:19.867222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.867104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9" (OuterVolumeSpecName: "kube-api-access-qlns9") pod "dd29f881-dc3a-4372-a71f-74c6d0635920" (UID: "dd29f881-dc3a-4372-a71f-74c6d0635920"). InnerVolumeSpecName "kube-api-access-qlns9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:19.886421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.886390    2578 generic.go:356] "Generic (PLEG): container finished" podID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerID="fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55" exitCode=0
Sep 15 16:40:19.886525 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.886458    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"
Sep 15 16:40:19.886525 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.886457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerDied","Data":"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"}
Sep 15 16:40:19.886620 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.886568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr" event={"ID":"dd29f881-dc3a-4372-a71f-74c6d0635920","Type":"ContainerDied","Data":"c7b9f4a7a9a3815e3950936b54584341ad7077cf024a04fba5cb8a723096df48"}
Sep 15 16:40:19.886620 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.886610    2578 scope.go:117] "RemoveContainer" containerID="3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"
Sep 15 16:40:19.896365 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.896342    2578 scope.go:117] "RemoveContainer" containerID="fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"
Sep 15 16:40:19.903977 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.903959    2578 scope.go:117] "RemoveContainer" containerID="f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"
Sep 15 16:40:19.909854 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.909831    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:40:19.912540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.912518    2578 scope.go:117] "RemoveContainer" containerID="3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"
Sep 15 16:40:19.912819 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:40:19.912801    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918\": container with ID starting with 3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918 not found: ID does not exist" containerID="3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"
Sep 15 16:40:19.912884 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.912827    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918"} err="failed to get container status \"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918\": rpc error: code = NotFound desc = could not find container \"3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918\": container with ID starting with 3f8f313ee58fc7d9b809899344147f1a079f46b8e78b5c49d7e31700a37a2918 not found: ID does not exist"
Sep 15 16:40:19.912884 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.912844    2578 scope.go:117] "RemoveContainer" containerID="fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"
Sep 15 16:40:19.913140 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:40:19.913119    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55\": container with ID starting with fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55 not found: ID does not exist" containerID="fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"
Sep 15 16:40:19.913209 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.913179    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55"} err="failed to get container status \"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55\": rpc error: code = NotFound desc = could not find container \"fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55\": container with ID starting with fe59736cc0d52c01b404fd3b7bc575ca2919c1ab417b1cf017b04e3410653f55 not found: ID does not exist"
Sep 15 16:40:19.913279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.913206    2578 scope.go:117] "RemoveContainer" containerID="f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"
Sep 15 16:40:19.913459 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:40:19.913441    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52\": container with ID starting with f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52 not found: ID does not exist" containerID="f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"
Sep 15 16:40:19.913508 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.913461    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52"} err="failed to get container status \"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52\": rpc error: code = NotFound desc = could not find container \"f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52\": container with ID starting with f2676980e7b7ab21c0d4e0dedce2f9b6f39392df6049412ce74ae55a1c516c52 not found: ID does not exist"
Sep 15 16:40:19.914052 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.914034    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bbdxr"]
Sep 15 16:40:19.966043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.966014    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd29f881-dc3a-4372-a71f-74c6d0635920-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:19.966043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.966037    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:19.966043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.966047    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlns9\" (UniqueName: \"kubernetes.io/projected/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-api-access-qlns9\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:19.966248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.966076    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd29f881-dc3a-4372-a71f-74c6d0635920-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:19.966248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:19.966086    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd29f881-dc3a-4372-a71f-74c6d0635920-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:40:20.228151 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228110    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:40:20.228550 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228531    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kube-rbac-proxy"
Sep 15 16:40:20.228649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228553    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kube-rbac-proxy"
Sep 15 16:40:20.228649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy"
Sep 15 16:40:20.228649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy"
Sep 15 16:40:20.228649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228637    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="storage-initializer"
Sep 15 16:40:20.228649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228647    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="storage-initializer"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228665    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228673    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228691    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="storage-initializer"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228700    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="storage-initializer"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228717    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228759    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228770    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228908    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kube-rbac-proxy"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228924    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228934    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kserve-container"
Sep 15 16:40:20.228942 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228942    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" containerName="kube-rbac-proxy"
Sep 15 16:40:20.229535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.228956    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67feb104-406a-45ea-95c2-f590f94c3998" containerName="agent"
Sep 15 16:40:20.234021 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.233999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.236255 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.236240    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 15 16:40:20.236347 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.236258    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 15 16:40:20.241501 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.241483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:40:20.268473 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.268434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.268639 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.268481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.268639 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.268550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.268712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.268634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.268712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.268666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8c55\" (UniqueName: \"kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369315 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8c55\" (UniqueName: \"kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369560 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:40:20.369445    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 15 16:40:20.369560 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.369560 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:40:20.369523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls podName:db95aacd-968d-4f14-8159-e89f7ebaef90 nodeName:}" failed. No retries permitted until 2026-09-15 16:40:20.869501749 +0000 UTC m=+792.256140995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 15 16:40:20.369915 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.369896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.370301 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.370280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.370340 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.370280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.377856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.377837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8c55\" (UniqueName: \"kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.872824 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.872784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:20.875385 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:20.875364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:21.145135 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:21.145107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:21.154757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:21.154700    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd29f881-dc3a-4372-a71f-74c6d0635920" path="/var/lib/kubelet/pods/dd29f881-dc3a-4372-a71f-74c6d0635920/volumes"
Sep 15 16:40:21.420767 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:21.420549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:40:21.423121 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:40:21.423098    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb95aacd_968d_4f14_8159_e89f7ebaef90.slice/crio-48bd0d0f1ac7d096561b602309fba524db9dc5650c7a694f5b8ee8de9bac2295 WatchSource:0}: Error finding container 48bd0d0f1ac7d096561b602309fba524db9dc5650c7a694f5b8ee8de9bac2295: Status 404 returned error can't find the container with id 48bd0d0f1ac7d096561b602309fba524db9dc5650c7a694f5b8ee8de9bac2295
Sep 15 16:40:21.900040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:21.900007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerStarted","Data":"512595fb8d7f33c58e915f4c2a55e1797e11641d836353dfe0213333f628674d"}
Sep 15 16:40:21.900428 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:21.900064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerStarted","Data":"48bd0d0f1ac7d096561b602309fba524db9dc5650c7a694f5b8ee8de9bac2295"}
Sep 15 16:40:25.919749 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:25.919698    2578 generic.go:356] "Generic (PLEG): container finished" podID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerID="512595fb8d7f33c58e915f4c2a55e1797e11641d836353dfe0213333f628674d" exitCode=0
Sep 15 16:40:25.920135 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:25.919774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerDied","Data":"512595fb8d7f33c58e915f4c2a55e1797e11641d836353dfe0213333f628674d"}
Sep 15 16:40:26.926692 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.926654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerStarted","Data":"f8568072c14b7c2fab1567a6d75127289f981b28f90ae58113e23af145175da0"}
Sep 15 16:40:26.927114 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.926721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerStarted","Data":"35701b660ffe752a43cd53606305586491d76dfd10b3c529aed096196a90aa99"}
Sep 15 16:40:26.927114 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.927059    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:26.927223 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.927202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:26.928613 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.928587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:26.945692 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:26.945646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podStartSLOduration=2.985338593 podStartE2EDuration="6.945631345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.960292752s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:26.944508629 +0000 UTC m=+798.331147898" watchObservedRunningTime="2026-09-15 16:40:26.945631345 +0000 UTC m=+798.332270614"
Sep 15 16:40:27.847065 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:27.847016    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:27.847486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:27.847459    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:27.931448 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:27.931414    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:28.936515 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:28.936468    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:33.941137 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:33.941053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:40:33.941758 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:33.941695    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:37.846509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:37.846470    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:37.847040 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:37.847017    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:43.941629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:43.941582    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:47.846389 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:47.846346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:47.846816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:47.846600    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:53.941663 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:53.941621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:40:57.846780 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:57.846717    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:40:57.847261 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:40:57.847187    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:41:03.942201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:03.942158    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:41:07.846449 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:07.846413    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:41:07.846973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:07.846946    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:41:13.941745 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:13.941705    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:41:17.846284 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:17.846235    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:41:17.846712 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:17.846649    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:41:23.942286 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:23.942247    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:41:27.847003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:27.846969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:41:27.847423 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:27.847060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:41:33.942569 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:33.942527    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:41:39.915046 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:39.915016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-4d6ld_e5163354-4370-4769-a1cb-174ec4b58675/kserve-container/0.log"
Sep 15 16:41:40.049600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.049568    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:41:40.049930 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.049901    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" containerID="cri-o://bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d" gracePeriod=30
Sep 15 16:41:40.050057 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.049937    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" containerID="cri-o://c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967" gracePeriod=30
Sep 15 16:41:40.050057 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.049950    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" containerID="cri-o://c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf" gracePeriod=30
Sep 15 16:41:40.081469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.081437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:41:40.081687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.081655    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kserve-container" containerID="cri-o://3db266e2d5d485386bd063e2386b06ab67d350a5b74de75ade7cfdbf2050e541" gracePeriod=30
Sep 15 16:41:40.081847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.081783    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kube-rbac-proxy" containerID="cri-o://aae6b5ef3e07ba3fa2803e53a7846b501bb04a10c40f9e3bb62ff7762b477d6a" gracePeriod=30
Sep 15 16:41:40.202037 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.202010    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:41:40.205629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.205608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.208075 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.208024    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 15 16:41:40.208193 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.208055    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 15 16:41:40.217409 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.217371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:41:40.249762 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.249700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.249929 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.249868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz256\" (UniqueName: \"kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.249977 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.249959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.250013 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.249990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.250013 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.250008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.287691 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.287659    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5163354-4370-4769-a1cb-174ec4b58675" containerID="aae6b5ef3e07ba3fa2803e53a7846b501bb04a10c40f9e3bb62ff7762b477d6a" exitCode=2
Sep 15 16:41:40.287691 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.287687    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5163354-4370-4769-a1cb-174ec4b58675" containerID="3db266e2d5d485386bd063e2386b06ab67d350a5b74de75ade7cfdbf2050e541" exitCode=2
Sep 15 16:41:40.287930 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.287750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerDied","Data":"aae6b5ef3e07ba3fa2803e53a7846b501bb04a10c40f9e3bb62ff7762b477d6a"}
Sep 15 16:41:40.287930 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.287828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerDied","Data":"3db266e2d5d485386bd063e2386b06ab67d350a5b74de75ade7cfdbf2050e541"}
Sep 15 16:41:40.290820 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.290792    2578 generic.go:356] "Generic (PLEG): container finished" podID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerID="c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf" exitCode=2
Sep 15 16:41:40.290966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.290865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerDied","Data":"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"}
Sep 15 16:41:40.332689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.332668    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:41:40.350421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.350387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.350421 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.350422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.350619 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.350442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.350619 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:41:40.350531    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 15 16:41:40.350619 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.350561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.350619 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:41:40.350587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls podName:8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b nodeName:}" failed. No retries permitted until 2026-09-15 16:41:40.850568402 +0000 UTC m=+872.237207648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-2h99t" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 15 16:41:40.350888 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.350685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz256\" (UniqueName: \"kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.351102 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.351076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.351228 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.351192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.351559 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.351534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.360844 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.360822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz256\" (UniqueName: \"kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.451920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.451836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls\") pod \"e5163354-4370-4769-a1cb-174ec4b58675\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") "
Sep 15 16:41:40.451920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.451902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config\") pod \"e5163354-4370-4769-a1cb-174ec4b58675\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") "
Sep 15 16:41:40.452157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.451954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvk9g\" (UniqueName: \"kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g\") pod \"e5163354-4370-4769-a1cb-174ec4b58675\" (UID: \"e5163354-4370-4769-a1cb-174ec4b58675\") "
Sep 15 16:41:40.452265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.452239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5163354-4370-4769-a1cb-174ec4b58675" (UID: "e5163354-4370-4769-a1cb-174ec4b58675"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:41:40.454087 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.454064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g" (OuterVolumeSpecName: "kube-api-access-zvk9g") pod "e5163354-4370-4769-a1cb-174ec4b58675" (UID: "e5163354-4370-4769-a1cb-174ec4b58675"). InnerVolumeSpecName "kube-api-access-zvk9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:41:40.454175 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.454098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5163354-4370-4769-a1cb-174ec4b58675" (UID: "e5163354-4370-4769-a1cb-174ec4b58675"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:41:40.553113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.553083    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvk9g\" (UniqueName: \"kubernetes.io/projected/e5163354-4370-4769-a1cb-174ec4b58675-kube-api-access-zvk9g\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:40.553113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.553109    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5163354-4370-4769-a1cb-174ec4b58675-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:40.553113 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.553121    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5163354-4370-4769-a1cb-174ec4b58675-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:40.856304 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.856216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:40.858778 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:40.858759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2h99t\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:41.120134 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.120049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:41:41.298250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.298219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld" event={"ID":"e5163354-4370-4769-a1cb-174ec4b58675","Type":"ContainerDied","Data":"64e178a06cce01ca14b37b239b393ebebd1a7eddf35c30ce7a51a3cf6fdd1b99"}
Sep 15 16:41:41.298250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.298249    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"
Sep 15 16:41:41.298466 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.298287    2578 scope.go:117] "RemoveContainer" containerID="aae6b5ef3e07ba3fa2803e53a7846b501bb04a10c40f9e3bb62ff7762b477d6a"
Sep 15 16:41:41.307286 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.307251    2578 scope.go:117] "RemoveContainer" containerID="3db266e2d5d485386bd063e2386b06ab67d350a5b74de75ade7cfdbf2050e541"
Sep 15 16:41:41.315201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.315174    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:41:41.319442 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.319421    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-4d6ld"]
Sep 15 16:41:41.393198 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:41.393165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:41:41.396913 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:41:41.396883    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa9d893_ee97_4c86_90a2_2d3ffe5c6e9b.slice/crio-7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed WatchSource:0}: Error finding container 7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed: Status 404 returned error can't find the container with id 7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed
Sep 15 16:41:42.304827 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:42.304781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerStarted","Data":"6595931b7255e6a880975740653d23631cd6db6cc7f656ecb48cbabac5388123"}
Sep 15 16:41:42.305387 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:42.304895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerStarted","Data":"7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed"}
Sep 15 16:41:42.842530 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:42.842482    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:41:43.154686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:43.154651    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5163354-4370-4769-a1cb-174ec4b58675" path="/var/lib/kubelet/pods/e5163354-4370-4769-a1cb-174ec4b58675/volumes"
Sep 15 16:41:43.942560 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:43.942527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:41:45.322301 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:45.322268    2578 generic.go:356] "Generic (PLEG): container finished" podID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerID="bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d" exitCode=0
Sep 15 16:41:45.322681 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:45.322331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerDied","Data":"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"}
Sep 15 16:41:47.842907 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:47.842863    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:41:47.846241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:47.846214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:41:47.846609 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:47.846584    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:41:50.293160 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:50.293125    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:41:50.293650 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:50.293458    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" containerID="cri-o://35701b660ffe752a43cd53606305586491d76dfd10b3c529aed096196a90aa99" gracePeriod=30
Sep 15 16:41:50.293650 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:50.293576    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kube-rbac-proxy" containerID="cri-o://f8568072c14b7c2fab1567a6d75127289f981b28f90ae58113e23af145175da0" gracePeriod=30
Sep 15 16:41:51.354728 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:51.354698    2578 generic.go:356] "Generic (PLEG): container finished" podID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerID="f8568072c14b7c2fab1567a6d75127289f981b28f90ae58113e23af145175da0" exitCode=2
Sep 15 16:41:51.355110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:51.354773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerDied","Data":"f8568072c14b7c2fab1567a6d75127289f981b28f90ae58113e23af145175da0"}
Sep 15 16:41:52.843072 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:52.843028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:41:52.843450 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:52.843163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:41:53.936714 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:53.936671    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:41:53.942588 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:53.942560    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:41:56.383548 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.383471    2578 generic.go:356] "Generic (PLEG): container finished" podID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerID="35701b660ffe752a43cd53606305586491d76dfd10b3c529aed096196a90aa99" exitCode=0
Sep 15 16:41:56.383548 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.383521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerDied","Data":"35701b660ffe752a43cd53606305586491d76dfd10b3c529aed096196a90aa99"}
Sep 15 16:41:56.532159 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.532134    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:41:56.593086 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593053    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location\") pod \"db95aacd-968d-4f14-8159-e89f7ebaef90\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") "
Sep 15 16:41:56.593267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config\") pod \"db95aacd-968d-4f14-8159-e89f7ebaef90\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") "
Sep 15 16:41:56.593267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8c55\" (UniqueName: \"kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55\") pod \"db95aacd-968d-4f14-8159-e89f7ebaef90\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") "
Sep 15 16:41:56.593402 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert\") pod \"db95aacd-968d-4f14-8159-e89f7ebaef90\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") "
Sep 15 16:41:56.593475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") pod \"db95aacd-968d-4f14-8159-e89f7ebaef90\" (UID: \"db95aacd-968d-4f14-8159-e89f7ebaef90\") "
Sep 15 16:41:56.593475 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db95aacd-968d-4f14-8159-e89f7ebaef90" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:41:56.593579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db95aacd-968d-4f14-8159-e89f7ebaef90" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:41:56.593688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593669    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db95aacd-968d-4f14-8159-e89f7ebaef90-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:56.593688 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593691    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:56.593862 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.593664    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db95aacd-968d-4f14-8159-e89f7ebaef90" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:41:56.595320 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.595293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55" (OuterVolumeSpecName: "kube-api-access-q8c55") pod "db95aacd-968d-4f14-8159-e89f7ebaef90" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90"). InnerVolumeSpecName "kube-api-access-q8c55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:41:56.595551 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.595535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db95aacd-968d-4f14-8159-e89f7ebaef90" (UID: "db95aacd-968d-4f14-8159-e89f7ebaef90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:41:56.694103 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.694074    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8c55\" (UniqueName: \"kubernetes.io/projected/db95aacd-968d-4f14-8159-e89f7ebaef90-kube-api-access-q8c55\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:56.694103 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.694101    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db95aacd-968d-4f14-8159-e89f7ebaef90-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:56.694281 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:56.694112    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db95aacd-968d-4f14-8159-e89f7ebaef90-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:41:57.390665 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.390635    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"
Sep 15 16:41:57.391106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.390638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7" event={"ID":"db95aacd-968d-4f14-8159-e89f7ebaef90","Type":"ContainerDied","Data":"48bd0d0f1ac7d096561b602309fba524db9dc5650c7a694f5b8ee8de9bac2295"}
Sep 15 16:41:57.391106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.390809    2578 scope.go:117] "RemoveContainer" containerID="f8568072c14b7c2fab1567a6d75127289f981b28f90ae58113e23af145175da0"
Sep 15 16:41:57.399315 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.399292    2578 scope.go:117] "RemoveContainer" containerID="35701b660ffe752a43cd53606305586491d76dfd10b3c529aed096196a90aa99"
Sep 15 16:41:57.406719 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.406697    2578 scope.go:117] "RemoveContainer" containerID="512595fb8d7f33c58e915f4c2a55e1797e11641d836353dfe0213333f628674d"
Sep 15 16:41:57.409670 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.409649    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:41:57.415314 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.415259    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-vljs7"]
Sep 15 16:41:57.843019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.842932    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:41:57.846566 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.846537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:41:57.846950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:57.846926    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:41:59.154392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:41:59.154353    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" path="/var/lib/kubelet/pods/db95aacd-968d-4f14-8159-e89f7ebaef90/volumes"
Sep 15 16:42:02.842308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:02.842271    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:42:03.420027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:03.419989    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerID="6595931b7255e6a880975740653d23631cd6db6cc7f656ecb48cbabac5388123" exitCode=0
Sep 15 16:42:03.420209 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:03.420066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerDied","Data":"6595931b7255e6a880975740653d23631cd6db6cc7f656ecb48cbabac5388123"}
Sep 15 16:42:07.842988 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:07.842941    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 15 16:42:07.846465 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:07.846426    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 15 16:42:07.846601 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:07.846553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:42:07.846780 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:07.846730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:07.846891 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:07.846875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:42:10.255542 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.255517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:42:10.418837 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.418797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvn89\" (UniqueName: \"kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89\") pod \"94e1bb7b-e78e-4601-9614-a756e42d39db\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") "
Sep 15 16:42:10.419036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.418861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location\") pod \"94e1bb7b-e78e-4601-9614-a756e42d39db\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") "
Sep 15 16:42:10.419036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.418904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert\") pod \"94e1bb7b-e78e-4601-9614-a756e42d39db\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") "
Sep 15 16:42:10.419036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.418934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config\") pod \"94e1bb7b-e78e-4601-9614-a756e42d39db\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") "
Sep 15 16:42:10.419036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.418968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls\") pod \"94e1bb7b-e78e-4601-9614-a756e42d39db\" (UID: \"94e1bb7b-e78e-4601-9614-a756e42d39db\") "
Sep 15 16:42:10.419262 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.419214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94e1bb7b-e78e-4601-9614-a756e42d39db" (UID: "94e1bb7b-e78e-4601-9614-a756e42d39db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:42:10.419416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.419368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94e1bb7b-e78e-4601-9614-a756e42d39db" (UID: "94e1bb7b-e78e-4601-9614-a756e42d39db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:10.419416 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.419394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94e1bb7b-e78e-4601-9614-a756e42d39db" (UID: "94e1bb7b-e78e-4601-9614-a756e42d39db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:10.421576 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.421548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89" (OuterVolumeSpecName: "kube-api-access-wvn89") pod "94e1bb7b-e78e-4601-9614-a756e42d39db" (UID: "94e1bb7b-e78e-4601-9614-a756e42d39db"). InnerVolumeSpecName "kube-api-access-wvn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:42:10.421730 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.421707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94e1bb7b-e78e-4601-9614-a756e42d39db" (UID: "94e1bb7b-e78e-4601-9614-a756e42d39db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:42:10.465949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.465920    2578 generic.go:356] "Generic (PLEG): container finished" podID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerID="c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967" exitCode=0
Sep 15 16:42:10.466088 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.465983    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"
Sep 15 16:42:10.466088 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.465985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerDied","Data":"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"}
Sep 15 16:42:10.466171 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.466115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc" event={"ID":"94e1bb7b-e78e-4601-9614-a756e42d39db","Type":"ContainerDied","Data":"99782e8a42fd665923871fa9941c13c833e5ee658ef1ab64c041d596e289628e"}
Sep 15 16:42:10.466171 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.466145    2578 scope.go:117] "RemoveContainer" containerID="c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"
Sep 15 16:42:10.475593 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.475576    2578 scope.go:117] "RemoveContainer" containerID="c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"
Sep 15 16:42:10.484363 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.484344    2578 scope.go:117] "RemoveContainer" containerID="bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"
Sep 15 16:42:10.487996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.487974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:42:10.492524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.492503    2578 scope.go:117] "RemoveContainer" containerID="bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"
Sep 15 16:42:10.492830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.492804    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56c548f55d-p6rbc"]
Sep 15 16:42:10.499936 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.499907    2578 scope.go:117] "RemoveContainer" containerID="c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"
Sep 15 16:42:10.500210 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:42:10.500193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967\": container with ID starting with c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967 not found: ID does not exist" containerID="c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"
Sep 15 16:42:10.500261 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500219    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967"} err="failed to get container status \"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967\": rpc error: code = NotFound desc = could not find container \"c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967\": container with ID starting with c09329a14f6c70db7f8005da5b78c587baea1e75cc8e2889f2c2ef81df712967 not found: ID does not exist"
Sep 15 16:42:10.500261 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500238    2578 scope.go:117] "RemoveContainer" containerID="c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"
Sep 15 16:42:10.500502 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:42:10.500485    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf\": container with ID starting with c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf not found: ID does not exist" containerID="c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"
Sep 15 16:42:10.500569 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500514    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf"} err="failed to get container status \"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf\": rpc error: code = NotFound desc = could not find container \"c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf\": container with ID starting with c4665638ad19560342dc2ffbe3020624ced962d5a4ed7a98c8f1883d175a3acf not found: ID does not exist"
Sep 15 16:42:10.500569 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500535    2578 scope.go:117] "RemoveContainer" containerID="bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"
Sep 15 16:42:10.500825 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:42:10.500799    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d\": container with ID starting with bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d not found: ID does not exist" containerID="bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"
Sep 15 16:42:10.500864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500832    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d"} err="failed to get container status \"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d\": rpc error: code = NotFound desc = could not find container \"bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d\": container with ID starting with bd4e228b70707ceb926cd6de38199649383e6a06063720982e9cf50976f4956d not found: ID does not exist"
Sep 15 16:42:10.500864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.500846    2578 scope.go:117] "RemoveContainer" containerID="bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"
Sep 15 16:42:10.501090 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:42:10.501074    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe\": container with ID starting with bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe not found: ID does not exist" containerID="bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"
Sep 15 16:42:10.501134 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.501096    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe"} err="failed to get container status \"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe\": rpc error: code = NotFound desc = could not find container \"bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe\": container with ID starting with bc41fa1686537e775fbd5b3e0592b68d6a81d910fb2b20a8654174fc2f90cbbe not found: ID does not exist"
Sep 15 16:42:10.519664 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.519636    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:42:10.519664 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.519659    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:42:10.519836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.519675    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e1bb7b-e78e-4601-9614-a756e42d39db-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:42:10.519836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.519689    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvn89\" (UniqueName: \"kubernetes.io/projected/94e1bb7b-e78e-4601-9614-a756e42d39db-kube-api-access-wvn89\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:42:10.519836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:10.519700    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94e1bb7b-e78e-4601-9614-a756e42d39db-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:42:11.155795 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:42:11.155756    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" path="/var/lib/kubelet/pods/94e1bb7b-e78e-4601-9614-a756e42d39db/volumes"
Sep 15 16:44:03.026965 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:03.026936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerStarted","Data":"d5558249a80982bd55e0b29c0fd36df0bfa15613da47f4af51c42b5a8c56be05"}
Sep 15 16:44:03.027441 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:03.026990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerStarted","Data":"c1468d421f2edb5d18cc388cffb06485090ccf614eaa862097dbb9ff60fb5047"}
Sep 15 16:44:03.027441 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:03.027052    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:03.045906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:03.045854    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podStartSLOduration=3.049670367 podStartE2EDuration="2m23.045838405s" totalImagesPullingTime="1m58.731605421s" totalInitContainerRuntime="21.264562617s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:44:03.044658927 +0000 UTC m=+1014.431298195" watchObservedRunningTime="2026-09-15 16:44:03.045838405 +0000 UTC m=+1014.432477672"
Sep 15 16:44:04.031689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:04.031646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:04.033088 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:04.033058    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 15 16:44:05.037460 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:05.037407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 15 16:44:10.042076 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:10.042043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:10.042592 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:10.042563    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 15 16:44:20.042910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:20.042876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:36.764110 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:36.764075    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:44:36.764618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:36.764344    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container" containerID="cri-o://c1468d421f2edb5d18cc388cffb06485090ccf614eaa862097dbb9ff60fb5047" gracePeriod=30
Sep 15 16:44:36.764618 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:36.764384    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kube-rbac-proxy" containerID="cri-o://d5558249a80982bd55e0b29c0fd36df0bfa15613da47f4af51c42b5a8c56be05" gracePeriod=30
Sep 15 16:44:37.189386 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:37.189354    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerID="d5558249a80982bd55e0b29c0fd36df0bfa15613da47f4af51c42b5a8c56be05" exitCode=2
Sep 15 16:44:37.189608 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:37.189430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerDied","Data":"d5558249a80982bd55e0b29c0fd36df0bfa15613da47f4af51c42b5a8c56be05"}
Sep 15 16:44:38.196682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.196645    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerID="c1468d421f2edb5d18cc388cffb06485090ccf614eaa862097dbb9ff60fb5047" exitCode=0
Sep 15 16:44:38.197062 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.196727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerDied","Data":"c1468d421f2edb5d18cc388cffb06485090ccf614eaa862097dbb9ff60fb5047"}
Sep 15 16:44:38.197062 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.196837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t" event={"ID":"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b","Type":"ContainerDied","Data":"7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed"}
Sep 15 16:44:38.197062 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.196860    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7485816b2a8c3b189ee64a6b68bfc01f4fba8502c759cc92fe489b82566a09ed"
Sep 15 16:44:38.202788 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.202766    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:38.337713 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.337637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location\") pod \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") "
Sep 15 16:44:38.337713 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.337684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert\") pod \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") "
Sep 15 16:44:38.337955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.337717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config\") pod \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") "
Sep 15 16:44:38.337955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.337789    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") pod \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") "
Sep 15 16:44:38.337955 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.337865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz256\" (UniqueName: \"kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256\") pod \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\" (UID: \"8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b\") "
Sep 15 16:44:38.338137 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.338103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:38.338189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.338111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:38.339903 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.339870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:44:38.340059 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.340037    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256" (OuterVolumeSpecName: "kube-api-access-dz256") pod "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b"). InnerVolumeSpecName "kube-api-access-dz256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:44:38.342594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.342568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" (UID: "8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:44:38.438595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.438544    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:44:38.438595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.438589    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:44:38.438595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.438599    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:44:38.438595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.438610    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:44:38.438894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:38.438620    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz256\" (UniqueName: \"kubernetes.io/projected/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b-kube-api-access-dz256\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:44:39.201530 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:39.201499    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"
Sep 15 16:44:39.218568 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:39.218538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:44:39.224718 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:39.224693    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2h99t"]
Sep 15 16:44:41.154218 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.154187    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" path="/var/lib/kubelet/pods/8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b/volumes"
Sep 15 16:44:41.950755 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.950712    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:44:41.951066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kserve-container"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951068    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kserve-container"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951078    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951084    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951090    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951097    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951103    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="storage-initializer"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951109    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="storage-initializer"
Sep 15 16:44:41.951115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951117    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="storage-initializer"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="storage-initializer"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951130    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951134    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951144    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951159    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951173    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="storage-initializer"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951178    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="storage-initializer"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951187    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951192    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951197    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951202    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951208    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951213    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951257    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951263    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="agent"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951271    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951276    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951280    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e1bb7b-e78e-4601-9614-a756e42d39db" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951285    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kube-rbac-proxy"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951291    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa9d893-ee97-4c86-90a2-2d3ffe5c6e9b" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5163354-4370-4769-a1cb-174ec4b58675" containerName="kserve-container"
Sep 15 16:44:41.951354 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.951303    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db95aacd-968d-4f14-8159-e89f7ebaef90" containerName="kube-rbac-proxy"
Sep 15 16:44:41.953516 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.953497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:41.955750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.955700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 15 16:44:41.955750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.955707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 15 16:44:41.955920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.955846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:44:41.956001 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.955983    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:44:41.956634 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.956616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:44:41.956702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.956661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:44:41.956755 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.956698    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:44:41.963870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:41.963847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:44:42.070178 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.070144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.070469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.070209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.070469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.070281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.070469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.070310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.070469 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.070338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trnj\" (UniqueName: \"kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.170825 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.170785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.170832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trnj\" (UniqueName: \"kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.170914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.170970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.171036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.171204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171713 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.171689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.171783 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.171689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.173504 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.173471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.178616 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.178594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trnj\" (UniqueName: \"kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-bkcwh\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.264910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.264818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:44:42.540847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:42.540820    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:44:42.542552 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:44:42.542522    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd9cbd3_77c4_440c_8040_a1bf77fc9a84.slice/crio-ce1d05e0798d98da6696aaf0b7c09bfc844299de1cfc825dbb442ec269cf27c9 WatchSource:0}: Error finding container ce1d05e0798d98da6696aaf0b7c09bfc844299de1cfc825dbb442ec269cf27c9: Status 404 returned error can't find the container with id ce1d05e0798d98da6696aaf0b7c09bfc844299de1cfc825dbb442ec269cf27c9
Sep 15 16:44:43.221111 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:43.221072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerStarted","Data":"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"}
Sep 15 16:44:43.221488 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:44:43.221144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerStarted","Data":"ce1d05e0798d98da6696aaf0b7c09bfc844299de1cfc825dbb442ec269cf27c9"}
Sep 15 16:45:03.308431 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:03.308345    2578 generic.go:356] "Generic (PLEG): container finished" podID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerID="7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932" exitCode=0
Sep 15 16:45:03.308431 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:03.308406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerDied","Data":"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"}
Sep 15 16:45:04.315455 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:04.315417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerStarted","Data":"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"}
Sep 15 16:45:04.315842 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:04.315479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerStarted","Data":"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"}
Sep 15 16:45:04.315842 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:04.315679    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:45:04.335663 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:04.335619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" podStartSLOduration=2.993505554 podStartE2EDuration="23.335601564s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.34209601s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:45:04.335422054 +0000 UTC m=+1075.722061321" watchObservedRunningTime="2026-09-15 16:45:04.335601564 +0000 UTC m=+1075.722240831"
Sep 15 16:45:05.321259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:05.321225    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:45:11.335979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:45:11.335951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:46:40.152540 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:40.152460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:46:48.204706 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:48.204670    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:46:48.205293 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:48.204988    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kserve-container" containerID="cri-o://15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49" gracePeriod=30
Sep 15 16:46:48.205293 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:48.204988    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kube-rbac-proxy" containerID="cri-o://733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924" gracePeriod=30
Sep 15 16:46:48.798535 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:48.798504    2578 generic.go:356] "Generic (PLEG): container finished" podID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerID="733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924" exitCode=2
Sep 15 16:46:48.798730 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:48.798573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerDied","Data":"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"}
Sep 15 16:46:49.647568 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.647544    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:46:49.745874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.745791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls\") pod \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") "
Sep 15 16:46:49.745874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.745832    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert\") pod \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") "
Sep 15 16:46:49.745874 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.745862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location\") pod \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") "
Sep 15 16:46:49.746098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.745895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config\") pod \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") "
Sep 15 16:46:49.746098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.745958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4trnj\" (UniqueName: \"kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj\") pod \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\" (UID: \"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84\") "
Sep 15 16:46:49.746260 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.746234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" (UID: "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:49.746308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.746255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" (UID: "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:49.748136 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.748098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" (UID: "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:46:49.748252 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.748228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj" (OuterVolumeSpecName: "kube-api-access-4trnj") pod "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" (UID: "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84"). InnerVolumeSpecName "kube-api-access-4trnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:46:49.750631 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.750611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" (UID: "3bd9cbd3-77c4-440c-8040-a1bf77fc9a84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:46:49.804994 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.804966    2578 generic.go:356] "Generic (PLEG): container finished" podID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerID="15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49" exitCode=0
Sep 15 16:46:49.811224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.805298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerDied","Data":"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"}
Sep 15 16:46:49.811224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.805389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh" event={"ID":"3bd9cbd3-77c4-440c-8040-a1bf77fc9a84","Type":"ContainerDied","Data":"ce1d05e0798d98da6696aaf0b7c09bfc844299de1cfc825dbb442ec269cf27c9"}
Sep 15 16:46:49.811224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.805414    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"
Sep 15 16:46:49.811224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.805454    2578 scope.go:117] "RemoveContainer" containerID="733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"
Sep 15 16:46:49.820661 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.820644    2578 scope.go:117] "RemoveContainer" containerID="15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"
Sep 15 16:46:49.829185 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.829160    2578 scope.go:117] "RemoveContainer" containerID="7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"
Sep 15 16:46:49.830601 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.830575    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:46:49.833920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.833898    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-bkcwh"]
Sep 15 16:46:49.837025 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837000    2578 scope.go:117] "RemoveContainer" containerID="733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"
Sep 15 16:46:49.837308 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:46:49.837292    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924\": container with ID starting with 733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924 not found: ID does not exist" containerID="733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"
Sep 15 16:46:49.837359 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837316    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924"} err="failed to get container status \"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924\": rpc error: code = NotFound desc = could not find container \"733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924\": container with ID starting with 733a23b3919c3935fd5f0ba0ccf63e1e17f0d6ce8637a3ee3dd9e629b139d924 not found: ID does not exist"
Sep 15 16:46:49.837359 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837333    2578 scope.go:117] "RemoveContainer" containerID="15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"
Sep 15 16:46:49.837547 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:46:49.837528    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49\": container with ID starting with 15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49 not found: ID does not exist" containerID="15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"
Sep 15 16:46:49.837587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837553    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49"} err="failed to get container status \"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49\": rpc error: code = NotFound desc = could not find container \"15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49\": container with ID starting with 15ce6fb21a4fcbe583da04dac5e5a4cd239ff6f233b5a87f70c7fa0eaf993a49 not found: ID does not exist"
Sep 15 16:46:49.837587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837568    2578 scope.go:117] "RemoveContainer" containerID="7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"
Sep 15 16:46:49.837847 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:46:49.837829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932\": container with ID starting with 7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932 not found: ID does not exist" containerID="7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"
Sep 15 16:46:49.837895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.837852    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932"} err="failed to get container status \"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932\": rpc error: code = NotFound desc = could not find container \"7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932\": container with ID starting with 7f972a54d77344b0304b43d6b4c823ee30c8bdf0227928036d9ae1180fd04932 not found: ID does not exist"
Sep 15 16:46:49.847307 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.847287    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:46:49.847381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.847309    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:46:49.847381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.847319    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:46:49.847381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.847328    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:46:49.847381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:49.847336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trnj\" (UniqueName: \"kubernetes.io/projected/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84-kube-api-access-4trnj\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:46:51.154133 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:51.154101    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" path="/var/lib/kubelet/pods/3bd9cbd3-77c4-440c-8040-a1bf77fc9a84/volumes"
Sep 15 16:46:53.384508 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.384474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:46:53.384989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.384969    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kube-rbac-proxy"
Sep 15 16:46:53.385042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.384994    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kube-rbac-proxy"
Sep 15 16:46:53.385042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kserve-container"
Sep 15 16:46:53.385042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385031    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kserve-container"
Sep 15 16:46:53.385132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="storage-initializer"
Sep 15 16:46:53.385132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385049    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="storage-initializer"
Sep 15 16:46:53.385132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kserve-container"
Sep 15 16:46:53.385132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.385124    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd9cbd3-77c4-440c-8040-a1bf77fc9a84" containerName="kube-rbac-proxy"
Sep 15 16:46:53.388773 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.388724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.391259 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.391237    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 15 16:46:53.391405 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.391312    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:46:53.391486 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.391469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:46:53.391553 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.391535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:46:53.391722 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.391707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 15 16:46:53.392515 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.392497    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:46:53.392604 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.392527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:46:53.397398 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.397371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:46:53.477411 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.477374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgh7\" (UniqueName: \"kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.477566 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.477433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.477566 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.477485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.477566 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.477502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.477672 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.477590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgh7\" (UniqueName: \"kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:46:53.578502    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 15 16:46:53.578649 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:46:53.578559    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls podName:675df8c8-fe82-4ddf-876b-596286dd6edc nodeName:}" failed. No retries permitted until 2026-09-15 16:46:54.078542503 +0000 UTC m=+1185.465181749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 15 16:46:53.579010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.578880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.579128 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.579108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.579128 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.579119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:53.586906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:53.586888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgh7\" (UniqueName: \"kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:54.083246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.083203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:54.085686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.085664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:54.299922 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.299879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:46:54.572436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.572412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:46:54.574578 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:46:54.574548    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675df8c8_fe82_4ddf_876b_596286dd6edc.slice/crio-bfd9068b4cdc5d33bb99ef00b6cff3559c0bcefaf97a794b224491c5fc2e501e WatchSource:0}: Error finding container bfd9068b4cdc5d33bb99ef00b6cff3559c0bcefaf97a794b224491c5fc2e501e: Status 404 returned error can't find the container with id bfd9068b4cdc5d33bb99ef00b6cff3559c0bcefaf97a794b224491c5fc2e501e
Sep 15 16:46:54.836422 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.836384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerStarted","Data":"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"}
Sep 15 16:46:54.836630 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:46:54.836464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerStarted","Data":"bfd9068b4cdc5d33bb99ef00b6cff3559c0bcefaf97a794b224491c5fc2e501e"}
Sep 15 16:47:23.969875 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:23.969840    2578 generic.go:356] "Generic (PLEG): container finished" podID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerID="e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1" exitCode=0
Sep 15 16:47:23.970216 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:23.969911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerDied","Data":"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"}
Sep 15 16:47:24.978301 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:24.978266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerStarted","Data":"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"}
Sep 15 16:47:24.978792 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:24.978334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerStarted","Data":"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"}
Sep 15 16:47:24.978792 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:24.978530    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:47:24.999106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:24.999053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" podStartSLOduration=2.709173688 podStartE2EDuration="31.999037013s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.289863325s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:47:24.998278902 +0000 UTC m=+1216.384918165" watchObservedRunningTime="2026-09-15 16:47:24.999037013 +0000 UTC m=+1216.385676280"
Sep 15 16:47:25.983313 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:25.983284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:47:31.993633 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:31.993599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:47:44.990443 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:44.990407    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:47:44.997328 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:44.997294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.000048 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.000026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 15 16:47:45.000169 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.000047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 15 16:47:45.004981 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.004847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:47:45.116336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.116305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.116336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.116342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.116558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.116392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbmw\" (UniqueName: \"kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.116558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.116423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.116558 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.116445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.216978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217010 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217279 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:47:45.217143    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 15 16:47:45.217279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbmw\" (UniqueName: \"kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217279 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:47:45.217241    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls podName:d00b4601-62a3-42b5-b2e3-1f24e1c65453 nodeName:}" failed. No retries permitted until 2026-09-15 16:47:45.717218358 +0000 UTC m=+1237.103857617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls") pod "isvc-sklearn-predictor-5bb4fc798f-2nxm8" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 15 16:47:45.217510 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217510 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.217936 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.217903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.225546 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.225522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbmw\" (UniqueName: \"kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.722125 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.722071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.724674 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.724650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") pod \"isvc-sklearn-predictor-5bb4fc798f-2nxm8\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:45.908803 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:45.908762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:46.179771 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:46.179639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:47:46.182556 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:47:46.182526    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00b4601_62a3_42b5_b2e3_1f24e1c65453.slice/crio-3fcc353d423269d855606f9d14834ef2c85571b1056c435841ca602a95f29ec5 WatchSource:0}: Error finding container 3fcc353d423269d855606f9d14834ef2c85571b1056c435841ca602a95f29ec5: Status 404 returned error can't find the container with id 3fcc353d423269d855606f9d14834ef2c85571b1056c435841ca602a95f29ec5
Sep 15 16:47:47.084821 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:47.084791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerStarted","Data":"d2b02ca34bb7005201fe66844236aea1ca4b5a16fc5755ee4b6f096bcd82562c"}
Sep 15 16:47:47.085015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:47.084849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerStarted","Data":"3fcc353d423269d855606f9d14834ef2c85571b1056c435841ca602a95f29ec5"}
Sep 15 16:47:51.104836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:51.104803    2578 generic.go:356] "Generic (PLEG): container finished" podID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerID="d2b02ca34bb7005201fe66844236aea1ca4b5a16fc5755ee4b6f096bcd82562c" exitCode=0
Sep 15 16:47:51.105288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:51.104877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerDied","Data":"d2b02ca34bb7005201fe66844236aea1ca4b5a16fc5755ee4b6f096bcd82562c"}
Sep 15 16:47:52.112290 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:52.112253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerStarted","Data":"fd70fd87d80e9ec9ef9a3d8cc8814febff91d7bd8b123f7e621060047f41ec0f"}
Sep 15 16:47:52.112675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:52.112313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerStarted","Data":"00f9c3f4bbee1ca5ae01ed6f69a6d73236dcc82609c2fc5b2ad08f86c7f91313"}
Sep 15 16:47:52.112675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:52.112494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:52.132248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:52.132198    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podStartSLOduration=4.195249751 podStartE2EDuration="8.132183666s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936933915s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:47:52.130857575 +0000 UTC m=+1243.517496841" watchObservedRunningTime="2026-09-15 16:47:52.132183666 +0000 UTC m=+1243.518822933"
Sep 15 16:47:53.117223 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:53.117185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:53.118485 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:53.118455    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:47:54.121473 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:54.121430    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:47:59.125794 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:59.125764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:47:59.126257 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:47:59.126229    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:09.126360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:09.126307    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:10.804610 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:10.804580    2578 scope.go:117] "RemoveContainer" containerID="6595931b7255e6a880975740653d23631cd6db6cc7f656ecb48cbabac5388123"
Sep 15 16:48:19.127007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:19.126968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:29.126202 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:29.126165    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:39.126219 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:39.126177    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:49.126242 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:49.126196    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:56.153109 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:56.153075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:48:59.126292 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:48:59.126255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:49:07.192906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:07.192865    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 16:49:09.126952 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:09.126923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:49:10.242410 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:10.242377    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:49:10.242865 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:10.242620    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kserve-container" containerID="cri-o://1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433" gracePeriod=30
Sep 15 16:49:10.242865 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:10.242667    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kube-rbac-proxy" containerID="cri-o://028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4" gracePeriod=30
Sep 15 16:49:10.501938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:10.501842    2578 generic.go:356] "Generic (PLEG): container finished" podID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerID="028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4" exitCode=2
Sep 15 16:49:10.501938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:10.501915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerDied","Data":"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"}
Sep 15 16:49:11.690750 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.690709    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:49:11.778770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.778654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert\") pod \"675df8c8-fe82-4ddf-876b-596286dd6edc\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") "
Sep 15 16:49:11.778770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.778715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") pod \"675df8c8-fe82-4ddf-876b-596286dd6edc\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") "
Sep 15 16:49:11.778770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.778758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bgh7\" (UniqueName: \"kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7\") pod \"675df8c8-fe82-4ddf-876b-596286dd6edc\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") "
Sep 15 16:49:11.779049 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.778777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location\") pod \"675df8c8-fe82-4ddf-876b-596286dd6edc\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") "
Sep 15 16:49:11.779049 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.778791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config\") pod \"675df8c8-fe82-4ddf-876b-596286dd6edc\" (UID: \"675df8c8-fe82-4ddf-876b-596286dd6edc\") "
Sep 15 16:49:11.779236 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.779207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "675df8c8-fe82-4ddf-876b-596286dd6edc" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:11.779818 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.779321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "675df8c8-fe82-4ddf-876b-596286dd6edc" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:11.781036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.781006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "675df8c8-fe82-4ddf-876b-596286dd6edc" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:49:11.781126 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.781013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7" (OuterVolumeSpecName: "kube-api-access-4bgh7") pod "675df8c8-fe82-4ddf-876b-596286dd6edc" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc"). InnerVolumeSpecName "kube-api-access-4bgh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:49:11.783522 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.783499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "675df8c8-fe82-4ddf-876b-596286dd6edc" (UID: "675df8c8-fe82-4ddf-876b-596286dd6edc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:49:11.879782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.879758    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:11.879782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.879782    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675df8c8-fe82-4ddf-876b-596286dd6edc-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:11.879962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.879791    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgh7\" (UniqueName: \"kubernetes.io/projected/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-api-access-4bgh7\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:11.879962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.879801    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/675df8c8-fe82-4ddf-876b-596286dd6edc-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:11.879962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:11.879809    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675df8c8-fe82-4ddf-876b-596286dd6edc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:12.513256 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.513225    2578 generic.go:356] "Generic (PLEG): container finished" podID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerID="1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433" exitCode=0
Sep 15 16:49:12.513440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.513282    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"
Sep 15 16:49:12.513440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.513303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerDied","Data":"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"}
Sep 15 16:49:12.513440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.513363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf" event={"ID":"675df8c8-fe82-4ddf-876b-596286dd6edc","Type":"ContainerDied","Data":"bfd9068b4cdc5d33bb99ef00b6cff3559c0bcefaf97a794b224491c5fc2e501e"}
Sep 15 16:49:12.513440 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.513400    2578 scope.go:117] "RemoveContainer" containerID="028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"
Sep 15 16:49:12.522695 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.522679    2578 scope.go:117] "RemoveContainer" containerID="1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"
Sep 15 16:49:12.530250 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.530228    2578 scope.go:117] "RemoveContainer" containerID="e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"
Sep 15 16:49:12.535613 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.535592    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:49:12.537891 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.537864    2578 scope.go:117] "RemoveContainer" containerID="028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"
Sep 15 16:49:12.538248 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:49:12.538224    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4\": container with ID starting with 028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4 not found: ID does not exist" containerID="028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"
Sep 15 16:49:12.538366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.538267    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4"} err="failed to get container status \"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4\": rpc error: code = NotFound desc = could not find container \"028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4\": container with ID starting with 028be2079d24c67e7177d11c16345c39d243cb765899acd8210f15020505bbe4 not found: ID does not exist"
Sep 15 16:49:12.538366 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.538287    2578 scope.go:117] "RemoveContainer" containerID="1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"
Sep 15 16:49:12.538677 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:49:12.538631    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433\": container with ID starting with 1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433 not found: ID does not exist" containerID="1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"
Sep 15 16:49:12.538677 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.538660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433"} err="failed to get container status \"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433\": rpc error: code = NotFound desc = could not find container \"1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433\": container with ID starting with 1639afe552bb3d8a48c32dbb533a156d8e2491aad371d416afe105d84c31b433 not found: ID does not exist"
Sep 15 16:49:12.538880 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.538680    2578 scope.go:117] "RemoveContainer" containerID="e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"
Sep 15 16:49:12.539182 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:49:12.539130    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1\": container with ID starting with e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1 not found: ID does not exist" containerID="e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"
Sep 15 16:49:12.539182 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.539159    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1"} err="failed to get container status \"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1\": rpc error: code = NotFound desc = could not find container \"e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1\": container with ID starting with e28f0570221f4d387b681ce4e13f284890e7ff478be860bae935afd856a787f1 not found: ID does not exist"
Sep 15 16:49:12.541318 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:12.541297    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-trfgf"]
Sep 15 16:49:13.154455 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:13.154421    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" path="/var/lib/kubelet/pods/675df8c8-fe82-4ddf-876b-596286dd6edc/volumes"
Sep 15 16:49:15.065358 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:15.065317    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:49:15.065830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:15.065614    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" containerID="cri-o://00f9c3f4bbee1ca5ae01ed6f69a6d73236dcc82609c2fc5b2ad08f86c7f91313" gracePeriod=30
Sep 15 16:49:15.065830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:15.065661    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kube-rbac-proxy" containerID="cri-o://fd70fd87d80e9ec9ef9a3d8cc8814febff91d7bd8b123f7e621060047f41ec0f" gracePeriod=30
Sep 15 16:49:15.530515 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:15.530483    2578 generic.go:356] "Generic (PLEG): container finished" podID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerID="fd70fd87d80e9ec9ef9a3d8cc8814febff91d7bd8b123f7e621060047f41ec0f" exitCode=2
Sep 15 16:49:15.530698 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:15.530560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerDied","Data":"fd70fd87d80e9ec9ef9a3d8cc8814febff91d7bd8b123f7e621060047f41ec0f"}
Sep 15 16:49:19.121812 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.121768    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 15 16:49:19.126551 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.126516    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:49:19.555027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.554981    2578 generic.go:356] "Generic (PLEG): container finished" podID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerID="00f9c3f4bbee1ca5ae01ed6f69a6d73236dcc82609c2fc5b2ad08f86c7f91313" exitCode=0
Sep 15 16:49:19.555166 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.555019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerDied","Data":"00f9c3f4bbee1ca5ae01ed6f69a6d73236dcc82609c2fc5b2ad08f86c7f91313"}
Sep 15 16:49:19.610784 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.610760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:49:19.647562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config\") pod \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") "
Sep 15 16:49:19.647562 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tbmw\" (UniqueName: \"kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw\") pod \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") "
Sep 15 16:49:19.647781 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert\") pod \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") "
Sep 15 16:49:19.647781 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location\") pod \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") "
Sep 15 16:49:19.647781 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") pod \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\" (UID: \"d00b4601-62a3-42b5-b2e3-1f24e1c65453\") "
Sep 15 16:49:19.647962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d00b4601-62a3-42b5-b2e3-1f24e1c65453" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:19.647962 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d00b4601-62a3-42b5-b2e3-1f24e1c65453" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:49:19.648051 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.647978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d00b4601-62a3-42b5-b2e3-1f24e1c65453" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:19.649862 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.649832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d00b4601-62a3-42b5-b2e3-1f24e1c65453" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:49:19.649926 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.649871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw" (OuterVolumeSpecName: "kube-api-access-4tbmw") pod "d00b4601-62a3-42b5-b2e3-1f24e1c65453" (UID: "d00b4601-62a3-42b5-b2e3-1f24e1c65453"). InnerVolumeSpecName "kube-api-access-4tbmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:49:19.749053 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.749006    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:19.749053 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.749048    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbmw\" (UniqueName: \"kubernetes.io/projected/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kube-api-access-4tbmw\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:19.749053 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.749058    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d00b4601-62a3-42b5-b2e3-1f24e1c65453-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:19.749053 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.749067    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00b4601-62a3-42b5-b2e3-1f24e1c65453-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:19.749290 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:19.749076    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d00b4601-62a3-42b5-b2e3-1f24e1c65453-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:49:20.213229 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213194    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213533    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kserve-container"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213544    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kserve-container"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kube-rbac-proxy"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213558    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kube-rbac-proxy"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213567    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="storage-initializer"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213573    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="storage-initializer"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213580    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kube-rbac-proxy"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213585    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kube-rbac-proxy"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213595    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="storage-initializer"
Sep 15 16:49:20.213600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213600    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="storage-initializer"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213608    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213613    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213667    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kserve-container"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213677    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kube-rbac-proxy"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213683    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="675df8c8-fe82-4ddf-876b-596286dd6edc" containerName="kube-rbac-proxy"
Sep 15 16:49:20.214035 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.213690    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" containerName="kserve-container"
Sep 15 16:49:20.217001 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.216982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.219274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.219249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 15 16:49:20.219373 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.219260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 15 16:49:20.226658 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.226631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:49:20.252274 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.252239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.252423 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.252279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.252423 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.252303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.252527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.252416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.252527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.252469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmbc\" (UniqueName: \"kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353685 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.353635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.353702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmbc\" (UniqueName: \"kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.353772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.353801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353870 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.353819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.353870 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:49:20.353843    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 15 16:49:20.354121 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:49:20.353914    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls podName:64a4cfe8-8df6-4b72-87f9-57f59cb73a87 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:20.853892631 +0000 UTC m=+1332.240531902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 15 16:49:20.354261 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.354245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.354480 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.354462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.354557 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.354540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.362152 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.362129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmbc\" (UniqueName: \"kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.561581 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.561510    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"
Sep 15 16:49:20.561581 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.561524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8" event={"ID":"d00b4601-62a3-42b5-b2e3-1f24e1c65453","Type":"ContainerDied","Data":"3fcc353d423269d855606f9d14834ef2c85571b1056c435841ca602a95f29ec5"}
Sep 15 16:49:20.561786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.561596    2578 scope.go:117] "RemoveContainer" containerID="fd70fd87d80e9ec9ef9a3d8cc8814febff91d7bd8b123f7e621060047f41ec0f"
Sep 15 16:49:20.570480 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.570463    2578 scope.go:117] "RemoveContainer" containerID="00f9c3f4bbee1ca5ae01ed6f69a6d73236dcc82609c2fc5b2ad08f86c7f91313"
Sep 15 16:49:20.578081 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.578063    2578 scope.go:117] "RemoveContainer" containerID="d2b02ca34bb7005201fe66844236aea1ca4b5a16fc5755ee4b6f096bcd82562c"
Sep 15 16:49:20.583298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.583273    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:49:20.587245 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.587227    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bb4fc798f-2nxm8"]
Sep 15 16:49:20.857904 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.857813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:20.860335 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:20.860310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:21.129129 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:21.129038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:49:21.155842 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:21.155810    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00b4601-62a3-42b5-b2e3-1f24e1c65453" path="/var/lib/kubelet/pods/d00b4601-62a3-42b5-b2e3-1f24e1c65453/volumes"
Sep 15 16:49:21.404150 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:21.404118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:49:21.408187 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:49:21.408157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a4cfe8_8df6_4b72_87f9_57f59cb73a87.slice/crio-2f5969c2a284d0e99a84c28b114e9da89755f3a35aa0eefe51d94a86000288bc WatchSource:0}: Error finding container 2f5969c2a284d0e99a84c28b114e9da89755f3a35aa0eefe51d94a86000288bc: Status 404 returned error can't find the container with id 2f5969c2a284d0e99a84c28b114e9da89755f3a35aa0eefe51d94a86000288bc
Sep 15 16:49:21.569533 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:21.569498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerStarted","Data":"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"}
Sep 15 16:49:21.569731 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:21.569559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerStarted","Data":"2f5969c2a284d0e99a84c28b114e9da89755f3a35aa0eefe51d94a86000288bc"}
Sep 15 16:49:25.589715 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:25.589680    2578 generic.go:356] "Generic (PLEG): container finished" podID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerID="6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3" exitCode=0
Sep 15 16:49:25.590073 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:25.589712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerDied","Data":"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"}
Sep 15 16:49:25.590835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:49:25.590819    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:50:10.828918 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:50:10.828883    2578 scope.go:117] "RemoveContainer" containerID="d5558249a80982bd55e0b29c0fd36df0bfa15613da47f4af51c42b5a8c56be05"
Sep 15 16:50:10.839181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:50:10.839159    2578 scope.go:117] "RemoveContainer" containerID="c1468d421f2edb5d18cc388cffb06485090ccf614eaa862097dbb9ff60fb5047"
Sep 15 16:51:30.272749 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:30.272711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerStarted","Data":"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"}
Sep 15 16:51:30.273316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:30.272796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerStarted","Data":"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"}
Sep 15 16:51:30.273316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:30.272833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:51:30.273316 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:30.272939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:51:30.301003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:30.300955    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podStartSLOduration=2.455606986 podStartE2EDuration="2m10.30093963s" totalImagesPullingTime="2m3.927482388s" totalInitContainerRuntime="3.917850256s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:30.299715163 +0000 UTC m=+1461.686354430" watchObservedRunningTime="2026-09-15 16:51:30.30093963 +0000 UTC m=+1461.687578962"
Sep 15 16:51:36.282197 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:51:36.282158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:52:06.386007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:06.385963    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 15 16:52:16.285243 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:16.285213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:52:20.371850 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:20.371818    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:52:20.372254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:20.372106    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container" containerID="cri-o://18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4" gracePeriod=30
Sep 15 16:52:20.372254 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:20.372117    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy" containerID="cri-o://5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc" gracePeriod=30
Sep 15 16:52:20.490365 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:20.490324    2578 generic.go:356] "Generic (PLEG): container finished" podID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerID="5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc" exitCode=2
Sep 15 16:52:20.490514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:20.490395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerDied","Data":"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"}
Sep 15 16:52:21.277395 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:21.277347    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 15 16:52:25.542296 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.542257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:25.545667 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.545645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.548014 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.547985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 15 16:52:25.548118 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.548081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 15 16:52:25.560136 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.560112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:25.621608 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.621572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52w4b\" (UniqueName: \"kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.621819 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.621619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.621819 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.621647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.621819 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.621684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.621819 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.621716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.722895 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.722865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52w4b\" (UniqueName: \"kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.722913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.722936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723144 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.723072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723144 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.723125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723303 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.723281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.723525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.723721 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.723700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.725580 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.725558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.730972 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.730948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52w4b\" (UniqueName: \"kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b\") pod \"isvc-sklearn-runtime-predictor-867b74777-qsnn8\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:25.862282 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:25.862200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:26.135719 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:26.135693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:26.137767 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:52:26.137723    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d23fc3f_4e00_4215_acf6_5e9279fc21e4.slice/crio-145210184da9a2ba6e485d0a30fba6b842d19a81efe333ed228ac39cf7fe930d WatchSource:0}: Error finding container 145210184da9a2ba6e485d0a30fba6b842d19a81efe333ed228ac39cf7fe930d: Status 404 returned error can't find the container with id 145210184da9a2ba6e485d0a30fba6b842d19a81efe333ed228ac39cf7fe930d
Sep 15 16:52:26.277234 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:26.277192    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 15 16:52:26.282624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:26.282596    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:52:26.517889 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:26.517810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerStarted","Data":"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"}
Sep 15 16:52:26.517889 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:26.517868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerStarted","Data":"145210184da9a2ba6e485d0a30fba6b842d19a81efe333ed228ac39cf7fe930d"}
Sep 15 16:52:27.507430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.507405    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:52:27.524315 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.524286    2578 generic.go:356] "Generic (PLEG): container finished" podID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerID="18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4" exitCode=0
Sep 15 16:52:27.524430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.524345    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"
Sep 15 16:52:27.524430 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.524366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerDied","Data":"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"}
Sep 15 16:52:27.524513 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.524441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj" event={"ID":"64a4cfe8-8df6-4b72-87f9-57f59cb73a87","Type":"ContainerDied","Data":"2f5969c2a284d0e99a84c28b114e9da89755f3a35aa0eefe51d94a86000288bc"}
Sep 15 16:52:27.524513 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.524491    2578 scope.go:117] "RemoveContainer" containerID="5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"
Sep 15 16:52:27.534394 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.534370    2578 scope.go:117] "RemoveContainer" containerID="18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"
Sep 15 16:52:27.539854 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.539834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqmbc\" (UniqueName: \"kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc\") pod \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") "
Sep 15 16:52:27.539979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.539894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") pod \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") "
Sep 15 16:52:27.539979 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.539929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert\") pod \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") "
Sep 15 16:52:27.540095 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.539982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config\") pod \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") "
Sep 15 16:52:27.540095 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.540012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location\") pod \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\" (UID: \"64a4cfe8-8df6-4b72-87f9-57f59cb73a87\") "
Sep 15 16:52:27.540433 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.540385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64a4cfe8-8df6-4b72-87f9-57f59cb73a87" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:52:27.540542 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.540394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64a4cfe8-8df6-4b72-87f9-57f59cb73a87" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:52:27.540542 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.540460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64a4cfe8-8df6-4b72-87f9-57f59cb73a87" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:52:27.542309 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.542280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc" (OuterVolumeSpecName: "kube-api-access-gqmbc") pod "64a4cfe8-8df6-4b72-87f9-57f59cb73a87" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87"). InnerVolumeSpecName "kube-api-access-gqmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:52:27.542401 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.542351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64a4cfe8-8df6-4b72-87f9-57f59cb73a87" (UID: "64a4cfe8-8df6-4b72-87f9-57f59cb73a87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:52:27.543525 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.543504    2578 scope.go:117] "RemoveContainer" containerID="6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"
Sep 15 16:52:27.552332 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.552314    2578 scope.go:117] "RemoveContainer" containerID="5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"
Sep 15 16:52:27.552609 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:27.552592    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc\": container with ID starting with 5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc not found: ID does not exist" containerID="5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"
Sep 15 16:52:27.552656 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.552616    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc"} err="failed to get container status \"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc\": rpc error: code = NotFound desc = could not find container \"5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc\": container with ID starting with 5e8e4a951e8123f15ed2ff9e12a664a34ed46acbaaa765a30504024c3db6eefc not found: ID does not exist"
Sep 15 16:52:27.552656 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.552633    2578 scope.go:117] "RemoveContainer" containerID="18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"
Sep 15 16:52:27.552929 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:27.552910    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4\": container with ID starting with 18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4 not found: ID does not exist" containerID="18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"
Sep 15 16:52:27.552992 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.552934    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4"} err="failed to get container status \"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4\": rpc error: code = NotFound desc = could not find container \"18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4\": container with ID starting with 18ef7524f8cc60f43a805a52c5d62b8024a78b1803ce73432b6d7bc50a67e3c4 not found: ID does not exist"
Sep 15 16:52:27.552992 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.552951    2578 scope.go:117] "RemoveContainer" containerID="6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"
Sep 15 16:52:27.553183 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:27.553167    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3\": container with ID starting with 6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3 not found: ID does not exist" containerID="6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"
Sep 15 16:52:27.553224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.553187    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3"} err="failed to get container status \"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3\": rpc error: code = NotFound desc = could not find container \"6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3\": container with ID starting with 6708c693575c4e4e8bd372b063001167d4e4042a3bfc1c8f32a0de6eaed0f7d3 not found: ID does not exist"
Sep 15 16:52:27.641066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.641029    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmbc\" (UniqueName: \"kubernetes.io/projected/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-api-access-gqmbc\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:27.641066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.641062    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:27.641291 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.641078    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:27.641291 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.641091    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:27.641291 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.641103    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64a4cfe8-8df6-4b72-87f9-57f59cb73a87-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:27.847815 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.847782    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:52:27.851251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:27.851226    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-q8ttj"]
Sep 15 16:52:29.160070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:29.154966    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" path="/var/lib/kubelet/pods/64a4cfe8-8df6-4b72-87f9-57f59cb73a87/volumes"
Sep 15 16:52:32.549334 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:32.549255    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerID="94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb" exitCode=0
Sep 15 16:52:32.549334 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:32.549305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerDied","Data":"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"}
Sep 15 16:52:33.556649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:33.556615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerStarted","Data":"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"}
Sep 15 16:52:33.557054 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:33.556690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerStarted","Data":"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"}
Sep 15 16:52:33.557054 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:33.556898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:33.574731 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:33.574679    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podStartSLOduration=2.4872164420000002 podStartE2EDuration="8.574663068s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.087446626s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:52:33.574181898 +0000 UTC m=+1524.960821165" watchObservedRunningTime="2026-09-15 16:52:33.574663068 +0000 UTC m=+1524.961302332"
Sep 15 16:52:34.562341 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:34.562308    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:34.563612 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:34.563581    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:52:35.568153 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:35.568116    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:52:40.572969 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:40.572938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:40.573455 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:40.573429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:52:50.573944 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:50.573907    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:57.477583 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:57.477538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:57.478023 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:57.477818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container" containerID="cri-o://c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf" gracePeriod=30
Sep 15 16:52:57.478023 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:57.477877    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kube-rbac-proxy" containerID="cri-o://dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3" gracePeriod=30
Sep 15 16:52:57.679289 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:57.679259    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerID="dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3" exitCode=2
Sep 15 16:52:57.679445 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:57.679321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerDied","Data":"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"}
Sep 15 16:52:58.622902 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.622877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:58.686081 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.686054    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerID="c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf" exitCode=0
Sep 15 16:52:58.686246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.686104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerDied","Data":"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"}
Sep 15 16:52:58.686246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.686110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"
Sep 15 16:52:58.686246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.686153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8" event={"ID":"4d23fc3f-4e00-4215-acf6-5e9279fc21e4","Type":"ContainerDied","Data":"145210184da9a2ba6e485d0a30fba6b842d19a81efe333ed228ac39cf7fe930d"}
Sep 15 16:52:58.686246 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.686176    2578 scope.go:117] "RemoveContainer" containerID="dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"
Sep 15 16:52:58.694484 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.694456    2578 scope.go:117] "RemoveContainer" containerID="c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"
Sep 15 16:52:58.701829 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.701812    2578 scope.go:117] "RemoveContainer" containerID="94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"
Sep 15 16:52:58.709321 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.709304    2578 scope.go:117] "RemoveContainer" containerID="dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"
Sep 15 16:52:58.709589 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:58.709570    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3\": container with ID starting with dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3 not found: ID does not exist" containerID="dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"
Sep 15 16:52:58.709632 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.709598    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3"} err="failed to get container status \"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3\": rpc error: code = NotFound desc = could not find container \"dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3\": container with ID starting with dff518448035dd9d87c3f3bff846829a9e6bc62763d986c7df67991ea2682fc3 not found: ID does not exist"
Sep 15 16:52:58.709632 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.709615    2578 scope.go:117] "RemoveContainer" containerID="c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"
Sep 15 16:52:58.709885 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:58.709866    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf\": container with ID starting with c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf not found: ID does not exist" containerID="c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"
Sep 15 16:52:58.709963 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.709895    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf"} err="failed to get container status \"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf\": rpc error: code = NotFound desc = could not find container \"c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf\": container with ID starting with c2a8237349e8615ea0356117f0ff6801548a5a2d076b8268793e6deff4acfadf not found: ID does not exist"
Sep 15 16:52:58.709963 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.709921    2578 scope.go:117] "RemoveContainer" containerID="94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"
Sep 15 16:52:58.710197 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:52:58.710179    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb\": container with ID starting with 94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb not found: ID does not exist" containerID="94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"
Sep 15 16:52:58.710253 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.710202    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb"} err="failed to get container status \"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb\": rpc error: code = NotFound desc = could not find container \"94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb\": container with ID starting with 94244c556a46d7859c5798905d56e0358776c7b331156592b3086e2c07fbb2eb not found: ID does not exist"
Sep 15 16:52:58.724509 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724489    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config\") pod \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") "
Sep 15 16:52:58.724603 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724589    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52w4b\" (UniqueName: \"kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b\") pod \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") "
Sep 15 16:52:58.724652 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location\") pod \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") "
Sep 15 16:52:58.724652 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls\") pod \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") "
Sep 15 16:52:58.724754 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert\") pod \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\" (UID: \"4d23fc3f-4e00-4215-acf6-5e9279fc21e4\") "
Sep 15 16:52:58.724900 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d23fc3f-4e00-4215-acf6-5e9279fc21e4" (UID: "4d23fc3f-4e00-4215-acf6-5e9279fc21e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:52:58.724995 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.724980    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:58.725112 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.725090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d23fc3f-4e00-4215-acf6-5e9279fc21e4" (UID: "4d23fc3f-4e00-4215-acf6-5e9279fc21e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:52:58.726864 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.726837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b" (OuterVolumeSpecName: "kube-api-access-52w4b") pod "4d23fc3f-4e00-4215-acf6-5e9279fc21e4" (UID: "4d23fc3f-4e00-4215-acf6-5e9279fc21e4"). InnerVolumeSpecName "kube-api-access-52w4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:52:58.726968 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.726871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d23fc3f-4e00-4215-acf6-5e9279fc21e4" (UID: "4d23fc3f-4e00-4215-acf6-5e9279fc21e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:52:58.751716 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.751681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d23fc3f-4e00-4215-acf6-5e9279fc21e4" (UID: "4d23fc3f-4e00-4215-acf6-5e9279fc21e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:52:58.826076 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.826048    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52w4b\" (UniqueName: \"kubernetes.io/projected/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kube-api-access-52w4b\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:58.826076 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.826072    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:58.826230 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.826082    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:58.826230 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:58.826092    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d23fc3f-4e00-4215-acf6-5e9279fc21e4-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:52:59.007554 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:59.007527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:59.010914 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:59.010889    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-867b74777-qsnn8"]
Sep 15 16:52:59.154164 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:52:59.154135    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" path="/var/lib/kubelet/pods/4d23fc3f-4e00-4215-acf6-5e9279fc21e4/volumes"
Sep 15 16:53:02.648933 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.648901    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649229    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649240    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649253    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="storage-initializer"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="storage-initializer"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649287    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649293    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649298 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649300    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="storage-initializer"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649305    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="storage-initializer"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649311    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649315    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649363    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649373    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a4cfe8-8df6-4b72-87f9-57f59cb73a87" containerName="kserve-container"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649379    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kube-rbac-proxy"
Sep 15 16:53:02.649587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.649392    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d23fc3f-4e00-4215-acf6-5e9279fc21e4" containerName="kserve-container"
Sep 15 16:53:02.654262 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.654244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.656891 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.656868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 15 16:53:02.657086 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.656890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:53:02.657086 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.657008    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 15 16:53:02.657240 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.657102    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:53:02.657324 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.657298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:53:02.657797 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.657778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:53:02.657890 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.657803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:53:02.660079 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.659946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.660079 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.660020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.660079 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.660052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkpd\" (UniqueName: \"kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.660275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.660090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.660275 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.660132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.662291 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.662272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:53:02.761046 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761046 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkpd\" (UniqueName: \"kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761300 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761300 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761300 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761663 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761850 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.761938 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.761920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.763723 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.763695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.769395 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.769366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkpd\" (UniqueName: \"kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:02.965872 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:02.965781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:03.240076 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:03.240053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:53:03.242160 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:53:03.242131    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f068ca_7be3_4216_8ed3_fdabe313a061.slice/crio-e59d90a1c232ae6d2590c5c8a74dffc6cacb0f03130eeaadb6a24734b70e2088 WatchSource:0}: Error finding container e59d90a1c232ae6d2590c5c8a74dffc6cacb0f03130eeaadb6a24734b70e2088: Status 404 returned error can't find the container with id e59d90a1c232ae6d2590c5c8a74dffc6cacb0f03130eeaadb6a24734b70e2088
Sep 15 16:53:03.711481 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:03.711447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerStarted","Data":"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"}
Sep 15 16:53:03.712003 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:03.711502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerStarted","Data":"e59d90a1c232ae6d2590c5c8a74dffc6cacb0f03130eeaadb6a24734b70e2088"}
Sep 15 16:53:07.734063 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:07.733975    2578 generic.go:356] "Generic (PLEG): container finished" podID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerID="4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6" exitCode=0
Sep 15 16:53:07.734063 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:07.734049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerDied","Data":"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"}
Sep 15 16:53:08.741069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:08.741031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerStarted","Data":"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"}
Sep 15 16:53:08.741465 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:08.741087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerStarted","Data":"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"}
Sep 15 16:53:08.741465 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:08.741307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:08.741589 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:08.741464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:08.761674 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:08.761617    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podStartSLOduration=2.6146885109999998 podStartE2EDuration="6.761601189s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.146912678s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:53:08.760073228 +0000 UTC m=+1560.146712495" watchObservedRunningTime="2026-09-15 16:53:08.761601189 +0000 UTC m=+1560.148240457"
Sep 15 16:53:14.751253 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:14.751222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:53:44.788347 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:44.788287    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 15 16:53:54.753630 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:53:54.753596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:54:02.719590 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:02.719506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:54:02.722248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:02.719763    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container" containerID="cri-o://ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e" gracePeriod=30
Sep 15 16:54:02.722248 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:02.719815    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy" containerID="cri-o://3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc" gracePeriod=30
Sep 15 16:54:02.985025 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:02.984945    2578 generic.go:356] "Generic (PLEG): container finished" podID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerID="3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc" exitCode=2
Sep 15 16:54:02.985166 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:02.985022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerDied","Data":"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"}
Sep 15 16:54:04.746227 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:04.746185    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 15 16:54:05.792945 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:05.792895    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 16:54:07.897527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:07.897484    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:54:07.901372 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:07.901349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:07.903949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:07.903930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 15 16:54:07.904047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:07.903935    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 15 16:54:07.910626 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:07.910604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:54:08.026785 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.026752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.026785 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.026790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.027034 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.026827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.027034 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.026854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.027034 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.026936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdh2\" (UniqueName: \"kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.127598 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.127567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.127598 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.127600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.127819 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.127671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdh2\" (UniqueName: \"kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.127866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.127813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.127919 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.127864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.128209 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.128182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.128393 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.128340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.128587 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.128476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.130223 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.130204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.135202 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.135183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdh2\" (UniqueName: \"kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2\") pod \"isvc-sklearn-v2-predictor-84b69c45b5-l8pmd\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.212308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.212221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:08.510231 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:08.510154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:54:08.512773 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:54:08.512723    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c21c7f6_770e_42bd_8b39_742c112cb892.slice/crio-380bb51fddeac00ac4b5a0156da73f6bc48525208f09c04dbf491bbbb0f18056 WatchSource:0}: Error finding container 380bb51fddeac00ac4b5a0156da73f6bc48525208f09c04dbf491bbbb0f18056: Status 404 returned error can't find the container with id 380bb51fddeac00ac4b5a0156da73f6bc48525208f09c04dbf491bbbb0f18056
Sep 15 16:54:09.012241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:09.012206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerStarted","Data":"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"}
Sep 15 16:54:09.012617 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:09.012271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerStarted","Data":"380bb51fddeac00ac4b5a0156da73f6bc48525208f09c04dbf491bbbb0f18056"}
Sep 15 16:54:09.746281 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:09.746236    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 15 16:54:09.963812 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:09.963784    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:54:10.018219 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.018147    2578 generic.go:356] "Generic (PLEG): container finished" podID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerID="ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e" exitCode=0
Sep 15 16:54:10.018219 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.018204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"
Sep 15 16:54:10.018645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.018235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerDied","Data":"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"}
Sep 15 16:54:10.018645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.018299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx" event={"ID":"05f068ca-7be3-4216-8ed3-fdabe313a061","Type":"ContainerDied","Data":"e59d90a1c232ae6d2590c5c8a74dffc6cacb0f03130eeaadb6a24734b70e2088"}
Sep 15 16:54:10.018645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.018327    2578 scope.go:117] "RemoveContainer" containerID="3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"
Sep 15 16:54:10.026731 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.026717    2578 scope.go:117] "RemoveContainer" containerID="ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"
Sep 15 16:54:10.034131 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.034117    2578 scope.go:117] "RemoveContainer" containerID="4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"
Sep 15 16:54:10.041424 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.041408    2578 scope.go:117] "RemoveContainer" containerID="3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"
Sep 15 16:54:10.041719 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:54:10.041699    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc\": container with ID starting with 3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc not found: ID does not exist" containerID="3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"
Sep 15 16:54:10.041804 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.041728    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc"} err="failed to get container status \"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc\": rpc error: code = NotFound desc = could not find container \"3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc\": container with ID starting with 3461cae0cd48c51e2588475905b8a0e40af937870728294287bf5cb704bcaafc not found: ID does not exist"
Sep 15 16:54:10.041804 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.041767    2578 scope.go:117] "RemoveContainer" containerID="ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"
Sep 15 16:54:10.042017 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:54:10.042000    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e\": container with ID starting with ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e not found: ID does not exist" containerID="ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"
Sep 15 16:54:10.042066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.042025    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e"} err="failed to get container status \"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e\": rpc error: code = NotFound desc = could not find container \"ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e\": container with ID starting with ff6cda3a985cecec1909b608a96480510f134cb066d6c4997eb530f8c0f0c48e not found: ID does not exist"
Sep 15 16:54:10.042066 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.042040    2578 scope.go:117] "RemoveContainer" containerID="4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"
Sep 15 16:54:10.042265 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:54:10.042249    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6\": container with ID starting with 4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6 not found: ID does not exist" containerID="4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"
Sep 15 16:54:10.042314 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.042269    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6"} err="failed to get container status \"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6\": rpc error: code = NotFound desc = could not find container \"4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6\": container with ID starting with 4d5496cd2819e7ad65e1de125ec171ebcb46b8f79b32c6fe161a2ddcdf379ac6 not found: ID does not exist"
Sep 15 16:54:10.047595 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.047581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls\") pod \"05f068ca-7be3-4216-8ed3-fdabe313a061\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") "
Sep 15 16:54:10.047661 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.047648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert\") pod \"05f068ca-7be3-4216-8ed3-fdabe313a061\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") "
Sep 15 16:54:10.047698 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.047672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location\") pod \"05f068ca-7be3-4216-8ed3-fdabe313a061\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") "
Sep 15 16:54:10.047755 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.047715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnkpd\" (UniqueName: \"kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd\") pod \"05f068ca-7be3-4216-8ed3-fdabe313a061\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") "
Sep 15 16:54:10.047822 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.047759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config\") pod \"05f068ca-7be3-4216-8ed3-fdabe313a061\" (UID: \"05f068ca-7be3-4216-8ed3-fdabe313a061\") "
Sep 15 16:54:10.048027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.048006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05f068ca-7be3-4216-8ed3-fdabe313a061" (UID: "05f068ca-7be3-4216-8ed3-fdabe313a061"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:54:10.048116 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.048071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05f068ca-7be3-4216-8ed3-fdabe313a061" (UID: "05f068ca-7be3-4216-8ed3-fdabe313a061"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:54:10.048163 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.048138    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05f068ca-7be3-4216-8ed3-fdabe313a061" (UID: "05f068ca-7be3-4216-8ed3-fdabe313a061"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:54:10.049624 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.049605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05f068ca-7be3-4216-8ed3-fdabe313a061" (UID: "05f068ca-7be3-4216-8ed3-fdabe313a061"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:54:10.049782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.049756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd" (OuterVolumeSpecName: "kube-api-access-hnkpd") pod "05f068ca-7be3-4216-8ed3-fdabe313a061" (UID: "05f068ca-7be3-4216-8ed3-fdabe313a061"). InnerVolumeSpecName "kube-api-access-hnkpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:54:10.149493 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.149459    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:54:10.149627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.149536    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05f068ca-7be3-4216-8ed3-fdabe313a061-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:54:10.149627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.149555    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkpd\" (UniqueName: \"kubernetes.io/projected/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-api-access-hnkpd\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:54:10.149627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.149566    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05f068ca-7be3-4216-8ed3-fdabe313a061-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:54:10.149627 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.149577    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05f068ca-7be3-4216-8ed3-fdabe313a061-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:54:10.340879 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.340850    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:54:10.343887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:10.343863    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-cz9tx"]
Sep 15 16:54:11.154381 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:11.154295    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" path="/var/lib/kubelet/pods/05f068ca-7be3-4216-8ed3-fdabe313a061/volumes"
Sep 15 16:54:13.032299 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:13.032271    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerID="e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601" exitCode=0
Sep 15 16:54:13.032676 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:13.032316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerDied","Data":"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"}
Sep 15 16:54:14.039032 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.038994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerStarted","Data":"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"}
Sep 15 16:54:14.039399 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.039082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerStarted","Data":"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"}
Sep 15 16:54:14.039399 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.039345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:14.039516 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.039494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:14.040523 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.040500    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:54:14.057920 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:14.057878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podStartSLOduration=3.3561906329999998 podStartE2EDuration="7.057865656s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.701675023s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:54:14.055845219 +0000 UTC m=+1625.442484485" watchObservedRunningTime="2026-09-15 16:54:14.057865656 +0000 UTC m=+1625.444504992"
Sep 15 16:54:15.043954 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:15.043917    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:54:20.048132 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:20.048100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:54:20.048559 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:20.048532    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:54:30.048807 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:30.048764    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:54:40.049361 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:40.049320    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:54:50.049202 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:54:50.049158    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:00.049444 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:00.049401    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:10.049123 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:10.049086    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:14.226693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.226658    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227050    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227069    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="storage-initializer"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227074    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="storage-initializer"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227087    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227093    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227161    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kserve-container"
Sep 15 16:55:14.227312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.227172    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f068ca-7be3-4216-8ed3-fdabe313a061" containerName="kube-rbac-proxy"
Sep 15 16:55:14.230498 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.230482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.232836 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.232811    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 15 16:55:14.232957 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.232843    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 15 16:55:14.240096 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.240072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:55:14.264175 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.264143    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:55:14.267777 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.267760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.269965 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.269945    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 15 16:55:14.276295 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.276269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:55:14.308922 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.308893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.309100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.308936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.309100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.308958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.309100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.309045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b94v\" (UniqueName: \"kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.309100 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.309081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410411 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksd9v\" (UniqueName: \"kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410600 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.410893 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b94v\" (UniqueName: \"kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410893 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.410893 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.410893 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:14.410861    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 15 16:55:14.411037 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:14.410931    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls podName:9bbb2c7a-9c31-42cd-b4ed-4254e379d740 nodeName:}" failed. No retries permitted until 2026-09-15 16:55:14.910910736 +0000 UTC m=+1686.297549995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls") pod "isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 15 16:55:14.411037 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.410984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.411167 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.411148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.411245 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.411230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.418617 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.418593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b94v\" (UniqueName: \"kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.511923 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.511843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.511923 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.511892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksd9v\" (UniqueName: \"kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.511923 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.511914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.512152 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.511959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.512152 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.512002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.512152 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:14.512097    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 15 16:55:14.512301 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:14.512184    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls podName:cb2e2ca9-c28a-4fe9-9de9-a7347435dd48 nodeName:}" failed. No retries permitted until 2026-09-15 16:55:15.012162156 +0000 UTC m=+1686.398801419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls") pod "isvc-canary-create-predictor-7d5b67cc88-gj5cf" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 15 16:55:14.512369 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.512352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.512520 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.512503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.512582 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.512565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.520318 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.520298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksd9v\" (UniqueName: \"kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:14.916042 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.916005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:14.918518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:14.918497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-jccsz\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:15.017007 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.016973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:15.019607 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.019585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-gj5cf\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:15.140869 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.140834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:15.178923 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.178853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:15.417142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.417115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:55:15.419069 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:55:15.419038    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbb2c7a_9c31_42cd_b4ed_4254e379d740.slice/crio-744f25028fdcd0675cbdcf5f0e02cbf9692c13ae2dd3a9036f16dc580d065c0a WatchSource:0}: Error finding container 744f25028fdcd0675cbdcf5f0e02cbf9692c13ae2dd3a9036f16dc580d065c0a: Status 404 returned error can't find the container with id 744f25028fdcd0675cbdcf5f0e02cbf9692c13ae2dd3a9036f16dc580d065c0a
Sep 15 16:55:15.464320 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:15.464300    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:55:15.466823 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:55:15.466799    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2e2ca9_c28a_4fe9_9de9_a7347435dd48.slice/crio-e61c4370e80e9ce22324c928d1ab912328a00135041480063d21657b73ae7e70 WatchSource:0}: Error finding container e61c4370e80e9ce22324c928d1ab912328a00135041480063d21657b73ae7e70: Status 404 returned error can't find the container with id e61c4370e80e9ce22324c928d1ab912328a00135041480063d21657b73ae7e70
Sep 15 16:55:16.305758 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:16.305695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerStarted","Data":"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"}
Sep 15 16:55:16.305954 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:16.305799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerStarted","Data":"744f25028fdcd0675cbdcf5f0e02cbf9692c13ae2dd3a9036f16dc580d065c0a"}
Sep 15 16:55:16.307805 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:16.307778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerStarted","Data":"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"}
Sep 15 16:55:16.307950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:16.307825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerStarted","Data":"e61c4370e80e9ce22324c928d1ab912328a00135041480063d21657b73ae7e70"}
Sep 15 16:55:19.325575 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:19.325532    2578 generic.go:356] "Generic (PLEG): container finished" podID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerID="6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55" exitCode=0
Sep 15 16:55:19.325975 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:19.325582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerDied","Data":"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"}
Sep 15 16:55:20.048652 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.048610    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:20.332541 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.332450    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerStarted","Data":"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"}
Sep 15 16:55:20.332541 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.332532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerStarted","Data":"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"}
Sep 15 16:55:20.333038 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.332774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:20.334061 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.334037    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerID="865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81" exitCode=0
Sep 15 16:55:20.334171 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.334110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerDied","Data":"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"}
Sep 15 16:55:20.351918 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:20.351863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podStartSLOduration=2.648060537 podStartE2EDuration="6.351841661s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.703781124s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:20.349227549 +0000 UTC m=+1691.735866817" watchObservedRunningTime="2026-09-15 16:55:20.351841661 +0000 UTC m=+1691.738480929"
Sep 15 16:55:21.149816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.149768    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:21.341095 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.341065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerStarted","Data":"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"}
Sep 15 16:55:21.341514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.341134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerStarted","Data":"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"}
Sep 15 16:55:21.341514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.341337    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:21.341615 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.341531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:21.342549 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.342524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:21.360051 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:21.360002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podStartSLOduration=3.261782941 podStartE2EDuration="7.359985655s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.098202714s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:21.359205047 +0000 UTC m=+1692.745844332" watchObservedRunningTime="2026-09-15 16:55:21.359985655 +0000 UTC m=+1692.746624924"
Sep 15 16:55:22.345801 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:22.345754    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:22.346290 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:22.345955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:22.346986 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:22.346964    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:23.350482 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:23.350442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:27.350226 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:27.350187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:55:27.350898 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:27.350871    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:28.356106 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:28.356079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:55:28.356592 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:28.356566    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:31.154556 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:31.154483    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:55:37.351286 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:37.351239    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:38.050808 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.050776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:55:38.051068 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.051022    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" containerID="cri-o://63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e" gracePeriod=30
Sep 15 16:55:38.051150 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.051046    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kube-rbac-proxy" containerID="cri-o://e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4" gracePeriod=30
Sep 15 16:55:38.356997 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.356914    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:38.423139 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.423104    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerID="e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4" exitCode=2
Sep 15 16:55:38.423301 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:38.423178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerDied","Data":"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"}
Sep 15 16:55:40.045150 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:40.045111    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 15 16:55:41.150182 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:41.150146    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:55:42.902412 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.902383    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:55:42.964986 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.964909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config\") pod \"0c21c7f6-770e-42bd-8b39-742c112cb892\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") "
Sep 15 16:55:42.964986 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.964958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert\") pod \"0c21c7f6-770e-42bd-8b39-742c112cb892\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") "
Sep 15 16:55:42.965189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fdh2\" (UniqueName: \"kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2\") pod \"0c21c7f6-770e-42bd-8b39-742c112cb892\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") "
Sep 15 16:55:42.965189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965053    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location\") pod \"0c21c7f6-770e-42bd-8b39-742c112cb892\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") "
Sep 15 16:55:42.965189 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls\") pod \"0c21c7f6-770e-42bd-8b39-742c112cb892\" (UID: \"0c21c7f6-770e-42bd-8b39-742c112cb892\") "
Sep 15 16:55:42.965586 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c21c7f6-770e-42bd-8b39-742c112cb892" (UID: "0c21c7f6-770e-42bd-8b39-742c112cb892"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:42.965707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c21c7f6-770e-42bd-8b39-742c112cb892" (UID: "0c21c7f6-770e-42bd-8b39-742c112cb892"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:42.965707 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c21c7f6-770e-42bd-8b39-742c112cb892" (UID: "0c21c7f6-770e-42bd-8b39-742c112cb892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:55:42.965832 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965803    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:55:42.965887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965835    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c21c7f6-770e-42bd-8b39-742c112cb892-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:55:42.965887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.965850    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c21c7f6-770e-42bd-8b39-742c112cb892-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:55:42.969782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.968148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2" (OuterVolumeSpecName: "kube-api-access-6fdh2") pod "0c21c7f6-770e-42bd-8b39-742c112cb892" (UID: "0c21c7f6-770e-42bd-8b39-742c112cb892"). InnerVolumeSpecName "kube-api-access-6fdh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:55:42.969782 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:42.968332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c21c7f6-770e-42bd-8b39-742c112cb892" (UID: "0c21c7f6-770e-42bd-8b39-742c112cb892"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:55:43.066995 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.066956    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdh2\" (UniqueName: \"kubernetes.io/projected/0c21c7f6-770e-42bd-8b39-742c112cb892-kube-api-access-6fdh2\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:55:43.066995 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.066988    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c21c7f6-770e-42bd-8b39-742c112cb892-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:55:43.222311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222227    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:55:43.222621 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222607    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="storage-initializer"
Sep 15 16:55:43.222675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222623    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="storage-initializer"
Sep 15 16:55:43.222675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222638    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kube-rbac-proxy"
Sep 15 16:55:43.222675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222643    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kube-rbac-proxy"
Sep 15 16:55:43.222675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222650    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container"
Sep 15 16:55:43.222675 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222655    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container"
Sep 15 16:55:43.222880 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kube-rbac-proxy"
Sep 15 16:55:43.222880 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.222731    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerName="kserve-container"
Sep 15 16:55:43.226171 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.226149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.228461 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.228435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 15 16:55:43.228575 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.228477    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 15 16:55:43.234621 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.234598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:55:43.269037 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.269006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.269170 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.269044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.269365 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.269327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.269491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.269460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brmb\" (UniqueName: \"kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.269491 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.269485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370041 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370235 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370235 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370235 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brmb\" (UniqueName: \"kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370235 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:43.370143    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 15 16:55:43.370235 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:43.370220    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls podName:d8064ab0-4cf4-4c97-b838-4fcdb91f1672 nodeName:}" failed. No retries permitted until 2026-09-15 16:55:43.87019739 +0000 UTC m=+1715.256836635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 15 16:55:43.370524 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370789 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.370839 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.370785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.380508 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.380486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brmb\" (UniqueName: \"kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.447494 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.447461    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c21c7f6-770e-42bd-8b39-742c112cb892" containerID="63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e" exitCode=0
Sep 15 16:55:43.447629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.447497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerDied","Data":"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"}
Sep 15 16:55:43.447629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.447532    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"
Sep 15 16:55:43.447629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.447555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd" event={"ID":"0c21c7f6-770e-42bd-8b39-742c112cb892","Type":"ContainerDied","Data":"380bb51fddeac00ac4b5a0156da73f6bc48525208f09c04dbf491bbbb0f18056"}
Sep 15 16:55:43.447629 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.447582    2578 scope.go:117] "RemoveContainer" containerID="e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"
Sep 15 16:55:43.456201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.456185    2578 scope.go:117] "RemoveContainer" containerID="63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"
Sep 15 16:55:43.464367 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.464332    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:55:43.467185 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.467166    2578 scope.go:117] "RemoveContainer" containerID="e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"
Sep 15 16:55:43.467689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.467663    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-84b69c45b5-l8pmd"]
Sep 15 16:55:43.475083 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475067    2578 scope.go:117] "RemoveContainer" containerID="e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"
Sep 15 16:55:43.475361 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:43.475341    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4\": container with ID starting with e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4 not found: ID does not exist" containerID="e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"
Sep 15 16:55:43.475419 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475371    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4"} err="failed to get container status \"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4\": rpc error: code = NotFound desc = could not find container \"e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4\": container with ID starting with e83bb435a370c16c17c02f2fef04c34f017a89d30e98f4398f3e174fa88827d4 not found: ID does not exist"
Sep 15 16:55:43.475419 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475392    2578 scope.go:117] "RemoveContainer" containerID="63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"
Sep 15 16:55:43.475654 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:43.475637    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e\": container with ID starting with 63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e not found: ID does not exist" containerID="63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"
Sep 15 16:55:43.475695 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e"} err="failed to get container status \"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e\": rpc error: code = NotFound desc = could not find container \"63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e\": container with ID starting with 63e73c9125c7bc3cbac1b46da5ba322faa2a9aeddedfad78fb9f95ce89b6ff1e not found: ID does not exist"
Sep 15 16:55:43.475695 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475676    2578 scope.go:117] "RemoveContainer" containerID="e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"
Sep 15 16:55:43.475950 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:55:43.475933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601\": container with ID starting with e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601 not found: ID does not exist" containerID="e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"
Sep 15 16:55:43.475989 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.475955    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601"} err="failed to get container status \"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601\": rpc error: code = NotFound desc = could not find container \"e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601\": container with ID starting with e8545b7978f5a2b8d23a730a912c3fa600cd1ebb66ff6e64e430518c42b64601 not found: ID does not exist"
Sep 15 16:55:43.874884 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.874800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:43.877292 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:43.877261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:44.136394 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:44.136357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:44.410567 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:44.410541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:55:44.413335 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:55:44.413297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8064ab0_4cf4_4c97_b838_4fcdb91f1672.slice/crio-13ec11a8b557587b4837d9d2809bbfa2b94228ec42dc7925302179824d739cfc WatchSource:0}: Error finding container 13ec11a8b557587b4837d9d2809bbfa2b94228ec42dc7925302179824d739cfc: Status 404 returned error can't find the container with id 13ec11a8b557587b4837d9d2809bbfa2b94228ec42dc7925302179824d739cfc
Sep 15 16:55:44.453209 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:44.453180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerStarted","Data":"13ec11a8b557587b4837d9d2809bbfa2b94228ec42dc7925302179824d739cfc"}
Sep 15 16:55:45.154950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:45.154919    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c21c7f6-770e-42bd-8b39-742c112cb892" path="/var/lib/kubelet/pods/0c21c7f6-770e-42bd-8b39-742c112cb892/volumes"
Sep 15 16:55:45.461321 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:45.461221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerStarted","Data":"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"}
Sep 15 16:55:47.351703 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:47.351661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:48.357321 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:48.357279    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:48.477615 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:48.477519    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerID="7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c" exitCode=0
Sep 15 16:55:48.477615 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:48.477590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerDied","Data":"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"}
Sep 15 16:55:49.484632 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.484598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerStarted","Data":"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"}
Sep 15 16:55:49.485039 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.484658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerStarted","Data":"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"}
Sep 15 16:55:49.485039 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.484980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:49.485138 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.485110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:49.486602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.486573    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:55:49.502922 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:49.502884    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podStartSLOduration=2.843520627 podStartE2EDuration="6.502871979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659351352s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:49.501572165 +0000 UTC m=+1720.888211432" watchObservedRunningTime="2026-09-15 16:55:49.502871979 +0000 UTC m=+1720.889511245"
Sep 15 16:55:50.492164 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:50.492117    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:55:55.497727 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:55.497692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:55:55.498271 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:55.498244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:55:57.351669 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:57.351631    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:55:58.357307 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:55:58.357269    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:56:05.498994 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:05.498953    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:07.350936 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:07.350897    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:56:08.357556 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:08.357517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:56:15.498966 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:15.498924    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:17.351559 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:17.351519    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:56:18.357463 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:18.357420    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:56:25.498273 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:25.498233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:27.350890 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:27.350851    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:56:28.356878 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:28.356838    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:56:35.498696 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:35.498660    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:37.352023 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.351993    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:56:37.467983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.467949    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:56:37.468201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.468178    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container" containerID="cri-o://8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae" gracePeriod=30
Sep 15 16:56:37.468279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.468228    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kube-rbac-proxy" containerID="cri-o://f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de" gracePeriod=30
Sep 15 16:56:37.469151 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.469014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:56:37.725921 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.725884    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerID="f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de" exitCode=2
Sep 15 16:56:37.726098 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:37.725952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerDied","Data":"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"}
Sep 15 16:56:38.351057 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:38.351019    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:56:42.217201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.217176    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:56:42.285288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") pod \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") "
Sep 15 16:56:42.285288 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config\") pod \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") "
Sep 15 16:56:42.285527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert\") pod \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") "
Sep 15 16:56:42.285527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksd9v\" (UniqueName: \"kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v\") pod \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") "
Sep 15 16:56:42.285527 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location\") pod \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\" (UID: \"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48\") "
Sep 15 16:56:42.285715 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:42.285818 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.285711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:42.286082 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.286054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:56:42.288181 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.288153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:56:42.288283 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.288186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v" (OuterVolumeSpecName: "kube-api-access-ksd9v") pod "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" (UID: "cb2e2ca9-c28a-4fe9-9de9-a7347435dd48"). InnerVolumeSpecName "kube-api-access-ksd9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:56:42.386949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.386915    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:42.386949 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.386944    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:42.387142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.386960    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksd9v\" (UniqueName: \"kubernetes.io/projected/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kube-api-access-ksd9v\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:42.387142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.386974    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:42.387142 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.386987    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:42.752506 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.752470    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerID="8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae" exitCode=0
Sep 15 16:56:42.752704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.752529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"
Sep 15 16:56:42.752704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.752549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerDied","Data":"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"}
Sep 15 16:56:42.752704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.752616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf" event={"ID":"cb2e2ca9-c28a-4fe9-9de9-a7347435dd48","Type":"ContainerDied","Data":"e61c4370e80e9ce22324c928d1ab912328a00135041480063d21657b73ae7e70"}
Sep 15 16:56:42.752704 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.752644    2578 scope.go:117] "RemoveContainer" containerID="f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"
Sep 15 16:56:42.761785 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.761769    2578 scope.go:117] "RemoveContainer" containerID="8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"
Sep 15 16:56:42.769431 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.769414    2578 scope.go:117] "RemoveContainer" containerID="865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"
Sep 15 16:56:42.773798 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.773776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:56:42.777360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.777339    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-gj5cf"]
Sep 15 16:56:42.777925 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.777911    2578 scope.go:117] "RemoveContainer" containerID="f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"
Sep 15 16:56:42.778211 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:42.778194    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de\": container with ID starting with f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de not found: ID does not exist" containerID="f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"
Sep 15 16:56:42.778272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.778218    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de"} err="failed to get container status \"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de\": rpc error: code = NotFound desc = could not find container \"f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de\": container with ID starting with f0f825cd3074292e7ce6a2136d6f31d92da57d2f7e8e154c3a2c0365b6e310de not found: ID does not exist"
Sep 15 16:56:42.778272 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.778234    2578 scope.go:117] "RemoveContainer" containerID="8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"
Sep 15 16:56:42.778471 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:42.778455    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae\": container with ID starting with 8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae not found: ID does not exist" containerID="8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"
Sep 15 16:56:42.784265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.778894    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae"} err="failed to get container status \"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae\": rpc error: code = NotFound desc = could not find container \"8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae\": container with ID starting with 8313bda7027665ef46995f9c456d3050b5edd12b0c6f90723712ddba28345bae not found: ID does not exist"
Sep 15 16:56:42.784265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.778962    2578 scope.go:117] "RemoveContainer" containerID="865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"
Sep 15 16:56:42.784265 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:42.782051    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81\": container with ID starting with 865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81 not found: ID does not exist" containerID="865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"
Sep 15 16:56:42.784265 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:42.782087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81"} err="failed to get container status \"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81\": rpc error: code = NotFound desc = could not find container \"865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81\": container with ID starting with 865b85651b2b636cdca06bf1f3e5dcfdf8aa0f74509bcc3b09da86318ace6d81 not found: ID does not exist"
Sep 15 16:56:43.154679 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:43.154647    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" path="/var/lib/kubelet/pods/cb2e2ca9-c28a-4fe9-9de9-a7347435dd48/volumes"
Sep 15 16:56:45.498549 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:45.498515    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:54.306340 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:54.306304    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:56:54.306910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:54.306550    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" containerID="cri-o://1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f" gracePeriod=30
Sep 15 16:56:54.306910 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:54.306602    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kube-rbac-proxy" containerID="cri-o://54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82" gracePeriod=30
Sep 15 16:56:54.810336 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:54.810292    2578 generic.go:356] "Generic (PLEG): container finished" podID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerID="54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82" exitCode=2
Sep 15 16:56:54.810503 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:54.810338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerDied","Data":"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"}
Sep 15 16:56:55.498918 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:55.498878    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:56:57.346543 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:57.346498    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 15 16:56:57.351408 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:57.351380    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 15 16:56:59.043133 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.043107    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:56:59.143432 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert\") pod \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") "
Sep 15 16:56:59.143432 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b94v\" (UniqueName: \"kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v\") pod \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") "
Sep 15 16:56:59.143432 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location\") pod \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") "
Sep 15 16:56:59.143720 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config\") pod \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") "
Sep 15 16:56:59.143720 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143512    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") pod \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\" (UID: \"9bbb2c7a-9c31-42cd-b4ed-4254e379d740\") "
Sep 15 16:56:59.143851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143808    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bbb2c7a-9c31-42cd-b4ed-4254e379d740" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:56:59.143851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9bbb2c7a-9c31-42cd-b4ed-4254e379d740" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:59.143983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.143959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bbb2c7a-9c31-42cd-b4ed-4254e379d740" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:59.145667 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.145642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v" (OuterVolumeSpecName: "kube-api-access-8b94v") pod "9bbb2c7a-9c31-42cd-b4ed-4254e379d740" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740"). InnerVolumeSpecName "kube-api-access-8b94v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:56:59.145769 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.145643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bbb2c7a-9c31-42cd-b4ed-4254e379d740" (UID: "9bbb2c7a-9c31-42cd-b4ed-4254e379d740"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:56:59.244570 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.244536    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.244570 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.244566    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b94v\" (UniqueName: \"kubernetes.io/projected/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-api-access-8b94v\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.244570 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.244578    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.244828 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.244587    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.244828 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.244596    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bbb2c7a-9c31-42cd-b4ed-4254e379d740-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.492760 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.492666    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 16:56:59.493267 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493246    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="storage-initializer"
Sep 15 16:56:59.493311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493273    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="storage-initializer"
Sep 15 16:56:59.493311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493299    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kube-rbac-proxy"
Sep 15 16:56:59.493374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kube-rbac-proxy"
Sep 15 16:56:59.493374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493324    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container"
Sep 15 16:56:59.493374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container"
Sep 15 16:56:59.493374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kube-rbac-proxy"
Sep 15 16:56:59.493374 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493358    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kube-rbac-proxy"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493386    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="storage-initializer"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="storage-initializer"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493408    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493507    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kserve-container"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493522    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kserve-container"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493531    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerName="kube-rbac-proxy"
Sep 15 16:56:59.493573 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.493544    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2e2ca9-c28a-4fe9-9de9-a7347435dd48" containerName="kube-rbac-proxy"
Sep 15 16:56:59.497776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.497760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.500220 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.500195    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 15 16:56:59.500220 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.500218    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 15 16:56:59.505218 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.505061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 16:56:59.546432 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.546405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.546602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.546465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.546602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.546539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcmd\" (UniqueName: \"kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.546602 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.546596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.546797 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.546652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.553645 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.553616    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:56:59.557084 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.557069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.559255 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.559234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 15 16:56:59.568823 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.568803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:56:59.647548 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpp8\" (UniqueName: \"kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.647765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.647765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.647765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.647765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcmd\" (UniqueName: \"kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.648019 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.647899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648222 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:59.648019    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 15 16:56:59.648222 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:59.648088    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls podName:5459b0bc-4086-4a08-b186-6194fa6dec9e nodeName:}" failed. No retries permitted until 2026-09-15 16:57:00.148068479 +0000 UTC m=+1791.534707738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls") pod "isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 15 16:56:59.648222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.648115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.648137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.648649 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.648629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.648687 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.648629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.658199 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.658179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcmd\" (UniqueName: \"kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:56:59.748594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.748520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpp8\" (UniqueName: \"kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.748594 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.748559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.748786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.748599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.748786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.748694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.748786 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.748727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.749036 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.749013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.749359 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.749295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.749359 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.749358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.751187 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.751168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.756479 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.756460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpp8\" (UniqueName: \"kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8\") pod \"isvc-canary-promote-predictor-6d6bc676fb-5t57j\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:56:59.839166 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.839132    2578 generic.go:356] "Generic (PLEG): container finished" podID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" containerID="1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f" exitCode=0
Sep 15 16:56:59.839351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.839193    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"
Sep 15 16:56:59.839351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.839215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerDied","Data":"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"}
Sep 15 16:56:59.839351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.839280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz" event={"ID":"9bbb2c7a-9c31-42cd-b4ed-4254e379d740","Type":"ContainerDied","Data":"744f25028fdcd0675cbdcf5f0e02cbf9692c13ae2dd3a9036f16dc580d065c0a"}
Sep 15 16:56:59.839351 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.839308    2578 scope.go:117] "RemoveContainer" containerID="54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"
Sep 15 16:56:59.847922 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.847902    2578 scope.go:117] "RemoveContainer" containerID="1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"
Sep 15 16:56:59.856694 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.856577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:56:59.856768 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.856747    2578 scope.go:117] "RemoveContainer" containerID="6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"
Sep 15 16:56:59.860075 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.860051    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-jccsz"]
Sep 15 16:56:59.864667 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.864650    2578 scope.go:117] "RemoveContainer" containerID="54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"
Sep 15 16:56:59.864948 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:59.864930    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82\": container with ID starting with 54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82 not found: ID does not exist" containerID="54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"
Sep 15 16:56:59.865024 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.864960    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82"} err="failed to get container status \"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82\": rpc error: code = NotFound desc = could not find container \"54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82\": container with ID starting with 54db0e2c9000d4ac47fc1e9aceb06b7c7f1aea90a10deafb7a5f50cbdd5bad82 not found: ID does not exist"
Sep 15 16:56:59.865024 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.864985    2578 scope.go:117] "RemoveContainer" containerID="1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"
Sep 15 16:56:59.865243 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:59.865226    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f\": container with ID starting with 1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f not found: ID does not exist" containerID="1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"
Sep 15 16:56:59.865311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.865249    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f"} err="failed to get container status \"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f\": rpc error: code = NotFound desc = could not find container \"1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f\": container with ID starting with 1c199f87f75c341986be20e9170d9e84f16b2df3690826d76121fa634cffae1f not found: ID does not exist"
Sep 15 16:56:59.865311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.865269    2578 scope.go:117] "RemoveContainer" containerID="6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"
Sep 15 16:56:59.865514 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:56:59.865495    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55\": container with ID starting with 6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55 not found: ID does not exist" containerID="6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"
Sep 15 16:56:59.865568 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.865522    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55"} err="failed to get container status \"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55\": rpc error: code = NotFound desc = could not find container \"6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55\": container with ID starting with 6f55ba75c6f01237bd37b41cb5abdaaf885f8cfad1d4f20a79a0935ce728ae55 not found: ID does not exist"
Sep 15 16:56:59.866659 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:56:59.866640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:57:00.137996 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.137972    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:57:00.140511 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:57:00.140484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93c0eb2_017d_4950_9d62_714f0ff6e4b3.slice/crio-5624957f540c16c8207748548275003de31328709c7b2528797cb6387af2b2c3 WatchSource:0}: Error finding container 5624957f540c16c8207748548275003de31328709c7b2528797cb6387af2b2c3: Status 404 returned error can't find the container with id 5624957f540c16c8207748548275003de31328709c7b2528797cb6387af2b2c3
Sep 15 16:57:00.152803 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.152775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:00.155237 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.155215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:00.409201 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.409096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:00.683235 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.683206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 16:57:00.685730 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:57:00.685696    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5459b0bc_4086_4a08_b186_6194fa6dec9e.slice/crio-33816ba9f3dd98b7240d9b31dfc383cf811b61a02bca73436e9c8b83d7150087 WatchSource:0}: Error finding container 33816ba9f3dd98b7240d9b31dfc383cf811b61a02bca73436e9c8b83d7150087: Status 404 returned error can't find the container with id 33816ba9f3dd98b7240d9b31dfc383cf811b61a02bca73436e9c8b83d7150087
Sep 15 16:57:00.847188 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.847145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerStarted","Data":"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"}
Sep 15 16:57:00.847379 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.847234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerStarted","Data":"33816ba9f3dd98b7240d9b31dfc383cf811b61a02bca73436e9c8b83d7150087"}
Sep 15 16:57:00.849947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.849915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerStarted","Data":"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"}
Sep 15 16:57:00.850052 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:00.849983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerStarted","Data":"5624957f540c16c8207748548275003de31328709c7b2528797cb6387af2b2c3"}
Sep 15 16:57:01.155156 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:01.155119    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bbb2c7a-9c31-42cd-b4ed-4254e379d740" path="/var/lib/kubelet/pods/9bbb2c7a-9c31-42cd-b4ed-4254e379d740/volumes"
Sep 15 16:57:04.873312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:04.873281    2578 generic.go:356] "Generic (PLEG): container finished" podID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerID="3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566" exitCode=0
Sep 15 16:57:04.873682 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:04.873356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerDied","Data":"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"}
Sep 15 16:57:04.875153 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:04.875123    2578 generic.go:356] "Generic (PLEG): container finished" podID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerID="ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e" exitCode=0
Sep 15 16:57:04.875232 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:04.875184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerDied","Data":"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"}
Sep 15 16:57:05.498875 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.498848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:57:05.882268 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.882172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerStarted","Data":"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"}
Sep 15 16:57:05.882689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.882293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerStarted","Data":"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"}
Sep 15 16:57:05.882689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.882621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:05.882689 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.882672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:05.884283 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.884255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:05.884667 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.884650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerStarted","Data":"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"}
Sep 15 16:57:05.884757 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.884698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerStarted","Data":"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"}
Sep 15 16:57:05.884933 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.884913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:57:05.901770 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.901704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podStartSLOduration=2.853572453 podStartE2EDuration="6.901689969s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.048117516s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:57:05.899057023 +0000 UTC m=+1797.285696301" watchObservedRunningTime="2026-09-15 16:57:05.901689969 +0000 UTC m=+1797.288329236"
Sep 15 16:57:05.915713 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:05.915660    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podStartSLOduration=3.104221411 podStartE2EDuration="6.915640495s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811419084s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:57:05.914592025 +0000 UTC m=+1797.301231292" watchObservedRunningTime="2026-09-15 16:57:05.915640495 +0000 UTC m=+1797.302279763"
Sep 15 16:57:06.891323 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:06.891272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:06.891861 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:06.891595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:57:06.892765 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:06.892719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:07.895822 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:07.895782    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:11.897363 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:11.897329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:57:11.897952 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:11.897925    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:12.900518 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:12.900490    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:57:12.901024 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:12.900999    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:13.302990 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:13.302906    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:57:13.303156 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:13.303134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" containerID="cri-o://4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40" gracePeriod=30
Sep 15 16:57:13.303226 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:13.303194    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kube-rbac-proxy" containerID="cri-o://973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637" gracePeriod=30
Sep 15 16:57:13.930514 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:13.930485    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerID="973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637" exitCode=2
Sep 15 16:57:13.930906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:13.930562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerDied","Data":"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"}
Sep 15 16:57:15.492457 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:15.492404    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 15 16:57:15.499080 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:15.499056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 15 16:57:18.252849 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.252821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:57:18.328487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config\") pod \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") "
Sep 15 16:57:18.328686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") pod \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") "
Sep 15 16:57:18.328686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert\") pod \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") "
Sep 15 16:57:18.328686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location\") pod \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") "
Sep 15 16:57:18.328686 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5brmb\" (UniqueName: \"kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb\") pod \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\" (UID: \"d8064ab0-4cf4-4c97-b838-4fcdb91f1672\") "
Sep 15 16:57:18.328967 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8064ab0-4cf4-4c97-b838-4fcdb91f1672" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:57:18.329025 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.328992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8064ab0-4cf4-4c97-b838-4fcdb91f1672" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:57:18.329069 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.329044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8064ab0-4cf4-4c97-b838-4fcdb91f1672" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:57:18.330947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.330922    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8064ab0-4cf4-4c97-b838-4fcdb91f1672" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:57:18.330947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.330926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb" (OuterVolumeSpecName: "kube-api-access-5brmb") pod "d8064ab0-4cf4-4c97-b838-4fcdb91f1672" (UID: "d8064ab0-4cf4-4c97-b838-4fcdb91f1672"). InnerVolumeSpecName "kube-api-access-5brmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:57:18.430148 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.430114    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5brmb\" (UniqueName: \"kubernetes.io/projected/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-api-access-5brmb\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:57:18.430308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.430155    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:57:18.430308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.430174    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:57:18.430308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.430192    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:57:18.430308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.430205    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8064ab0-4cf4-4c97-b838-4fcdb91f1672-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:57:18.956841 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.956799    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerID="4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40" exitCode=0
Sep 15 16:57:18.957022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.956867    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"
Sep 15 16:57:18.957022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.956869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerDied","Data":"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"}
Sep 15 16:57:18.957022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.956952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw" event={"ID":"d8064ab0-4cf4-4c97-b838-4fcdb91f1672","Type":"ContainerDied","Data":"13ec11a8b557587b4837d9d2809bbfa2b94228ec42dc7925302179824d739cfc"}
Sep 15 16:57:18.957022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.956981    2578 scope.go:117] "RemoveContainer" containerID="973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"
Sep 15 16:57:18.966188 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.966172    2578 scope.go:117] "RemoveContainer" containerID="4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"
Sep 15 16:57:18.973872 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.973854    2578 scope.go:117] "RemoveContainer" containerID="7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"
Sep 15 16:57:18.978626 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.978603    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:57:18.982702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.982677    2578 scope.go:117] "RemoveContainer" containerID="973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"
Sep 15 16:57:18.983046 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:57:18.983026    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637\": container with ID starting with 973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637 not found: ID does not exist" containerID="973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"
Sep 15 16:57:18.983129 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.983085    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637"} err="failed to get container status \"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637\": rpc error: code = NotFound desc = could not find container \"973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637\": container with ID starting with 973e852feb84ca9775525d64416fb4c90b08b0228ac19fff99453365cb063637 not found: ID does not exist"
Sep 15 16:57:18.983129 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.983111    2578 scope.go:117] "RemoveContainer" containerID="4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"
Sep 15 16:57:18.983407 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:57:18.983390    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40\": container with ID starting with 4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40 not found: ID does not exist" containerID="4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"
Sep 15 16:57:18.983479 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.983414    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40"} err="failed to get container status \"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40\": rpc error: code = NotFound desc = could not find container \"4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40\": container with ID starting with 4f3a6beac57df810e1199f967a1b7675d39e23b8b75d198173e104b560515d40 not found: ID does not exist"
Sep 15 16:57:18.983479 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.983432    2578 scope.go:117] "RemoveContainer" containerID="7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"
Sep 15 16:57:18.983794 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:57:18.983723    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c\": container with ID starting with 7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c not found: ID does not exist" containerID="7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"
Sep 15 16:57:18.983905 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.983800    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c"} err="failed to get container status \"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c\": rpc error: code = NotFound desc = could not find container \"7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c\": container with ID starting with 7a93cecfd9818f457da7cca36e405e1a98e51b5eae48e11b372c6bb8cf925c7c not found: ID does not exist"
Sep 15 16:57:18.984492 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:18.984473    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-894769d68-kqqkw"]
Sep 15 16:57:19.154851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:19.154817    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" path="/var/lib/kubelet/pods/d8064ab0-4cf4-4c97-b838-4fcdb91f1672/volumes"
Sep 15 16:57:21.898385 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:21.898346    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:22.901439 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:22.901397    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:31.898333 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:31.898297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:32.901930 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:32.901891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:41.898311 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:41.898275    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:42.901393 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:42.901347    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:57:51.898806 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:51.898760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:57:52.901952 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:57:52.901909    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:58:01.898708 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:01.898668    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:58:02.901429 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:02.901390    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:58:04.646843 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.646801    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:58:04.647368 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647349    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="storage-initializer"
Sep 15 16:58:04.647459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="storage-initializer"
Sep 15 16:58:04.647459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647385    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kube-rbac-proxy"
Sep 15 16:58:04.647459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kube-rbac-proxy"
Sep 15 16:58:04.647459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container"
Sep 15 16:58:04.647459 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647420    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container"
Sep 15 16:58:04.647716 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kserve-container"
Sep 15 16:58:04.647716 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.647572    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8064ab0-4cf4-4c97-b838-4fcdb91f1672" containerName="kube-rbac-proxy"
Sep 15 16:58:04.650994 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.650973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.653279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.653256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 15 16:58:04.653279 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.653271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 15 16:58:04.661332 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.661313    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:58:04.751312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.751278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgnq\" (UniqueName: \"kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.751487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.751354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.751487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.751395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.751487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.751415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.751487 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.751434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852186 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852360 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgnq\" (UniqueName: \"kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852816 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.852978 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.852959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.853027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.853001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.854894 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.854878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.864906 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.864881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgnq\" (UniqueName: \"kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:04.961494 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:04.961419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:05.238812 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:05.238783    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:58:05.240559 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:58:05.240528    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066a15f1_8bd5_4d5d_a14a_8c95625d6781.slice/crio-7613d9e85be56a35b43d584726443860594fd9f62c5c7c73a610792196cd4f38 WatchSource:0}: Error finding container 7613d9e85be56a35b43d584726443860594fd9f62c5c7c73a610792196cd4f38: Status 404 returned error can't find the container with id 7613d9e85be56a35b43d584726443860594fd9f62c5c7c73a610792196cd4f38
Sep 15 16:58:06.178253 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:06.178219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerStarted","Data":"711af1ab627a5385854a379b940fd660d562442f3732b749dd46bac512a6dbf4"}
Sep 15 16:58:06.178693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:06.178272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerStarted","Data":"7613d9e85be56a35b43d584726443860594fd9f62c5c7c73a610792196cd4f38"}
Sep 15 16:58:10.199550 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:10.199455    2578 generic.go:356] "Generic (PLEG): container finished" podID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerID="711af1ab627a5385854a379b940fd660d562442f3732b749dd46bac512a6dbf4" exitCode=0
Sep 15 16:58:10.199550 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:10.199530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerDied","Data":"711af1ab627a5385854a379b940fd660d562442f3732b749dd46bac512a6dbf4"}
Sep 15 16:58:10.200841 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:10.200823    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:58:11.897866 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:11.897824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 16:58:12.901820 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:12.901774    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 15 16:58:14.228156 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.228119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerStarted","Data":"7d4a5ba63ae46535cf74b7b941c0294f51c0aacd684f166a1ebd903477086e9c"}
Sep 15 16:58:14.228596 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.228180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerStarted","Data":"7fa6f490e851f8247b818a78eccc6a43d74102010adf62b5b54f178ddd40dd79"}
Sep 15 16:58:14.228596 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.228465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:14.228724 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.228635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:14.229863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.229841    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 15 16:58:14.248856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:14.248813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podStartSLOduration=1.8032801680000001 podStartE2EDuration="10.24879899s" totalImagesPullingTime="3.851839575s" totalInitContainerRuntime="4.593679247s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:58:14.246551635 +0000 UTC m=+1865.633190928" watchObservedRunningTime="2026-09-15 16:58:14.24879899 +0000 UTC m=+1865.635438260"
Sep 15 16:58:15.234162 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:15.234126    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 15 16:58:16.240070 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:16.240030    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 15 16:58:21.245678 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:21.245648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:21.246305 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:21.246190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 15 16:58:21.899364 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:21.899334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 16:58:22.008693 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.008657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:58:22.008946 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.008911    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container" containerID="cri-o://68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855" gracePeriod=30
Sep 15 16:58:22.009095 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.009001    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kube-rbac-proxy" containerID="cri-o://daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35" gracePeriod=30
Sep 15 16:58:22.009872 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.009854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:58:22.274950 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.274918    2578 generic.go:356] "Generic (PLEG): container finished" podID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerID="daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35" exitCode=2
Sep 15 16:58:22.275304 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.274993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerDied","Data":"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"}
Sep 15 16:58:22.896251 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:22.896208    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 15 16:58:26.846022 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.846000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:58:26.966830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.966718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert\") pod \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") "
Sep 15 16:58:26.966830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.966781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls\") pod \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") "
Sep 15 16:58:26.966830 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.966822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config\") pod \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") "
Sep 15 16:58:26.967115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.966857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location\") pod \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") "
Sep 15 16:58:26.967115 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.966944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wpp8\" (UniqueName: \"kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8\") pod \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\" (UID: \"a93c0eb2-017d-4950-9d62-714f0ff6e4b3\") "
Sep 15 16:58:26.967225 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.967119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a93c0eb2-017d-4950-9d62-714f0ff6e4b3" (UID: "a93c0eb2-017d-4950-9d62-714f0ff6e4b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:26.967225 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.967206    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:58:26.967308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.967218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a93c0eb2-017d-4950-9d62-714f0ff6e4b3" (UID: "a93c0eb2-017d-4950-9d62-714f0ff6e4b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:58:26.967308 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.967226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a93c0eb2-017d-4950-9d62-714f0ff6e4b3" (UID: "a93c0eb2-017d-4950-9d62-714f0ff6e4b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:26.969124 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.969101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8" (OuterVolumeSpecName: "kube-api-access-7wpp8") pod "a93c0eb2-017d-4950-9d62-714f0ff6e4b3" (UID: "a93c0eb2-017d-4950-9d62-714f0ff6e4b3"). InnerVolumeSpecName "kube-api-access-7wpp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:58:26.969226 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:26.969142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a93c0eb2-017d-4950-9d62-714f0ff6e4b3" (UID: "a93c0eb2-017d-4950-9d62-714f0ff6e4b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:58:27.067631 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.067601    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wpp8\" (UniqueName: \"kubernetes.io/projected/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-api-access-7wpp8\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:58:27.067631 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.067626    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:58:27.067847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.067642    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:58:27.067847 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.067656    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93c0eb2-017d-4950-9d62-714f0ff6e4b3-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:58:27.304496 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.304413    2578 generic.go:356] "Generic (PLEG): container finished" podID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerID="68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855" exitCode=0
Sep 15 16:58:27.304496 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.304479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"
Sep 15 16:58:27.304702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.304472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerDied","Data":"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"}
Sep 15 16:58:27.304702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.304613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j" event={"ID":"a93c0eb2-017d-4950-9d62-714f0ff6e4b3","Type":"ContainerDied","Data":"5624957f540c16c8207748548275003de31328709c7b2528797cb6387af2b2c3"}
Sep 15 16:58:27.304702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.304643    2578 scope.go:117] "RemoveContainer" containerID="daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"
Sep 15 16:58:27.313222 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.313205    2578 scope.go:117] "RemoveContainer" containerID="68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"
Sep 15 16:58:27.321835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.321541    2578 scope.go:117] "RemoveContainer" containerID="ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"
Sep 15 16:58:27.323496 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.323472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:58:27.328947 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.328920    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-5t57j"]
Sep 15 16:58:27.330848 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.330824    2578 scope.go:117] "RemoveContainer" containerID="daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"
Sep 15 16:58:27.331110 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:58:27.331091    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35\": container with ID starting with daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35 not found: ID does not exist" containerID="daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"
Sep 15 16:58:27.331168 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.331118    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35"} err="failed to get container status \"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35\": rpc error: code = NotFound desc = could not find container \"daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35\": container with ID starting with daffaf095537e8287d24250b4036903d76b415e0538f0f2a447bd31c05dc9b35 not found: ID does not exist"
Sep 15 16:58:27.331168 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.331134    2578 scope.go:117] "RemoveContainer" containerID="68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"
Sep 15 16:58:27.331395 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:58:27.331376    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855\": container with ID starting with 68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855 not found: ID does not exist" containerID="68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"
Sep 15 16:58:27.331436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.331401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855"} err="failed to get container status \"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855\": rpc error: code = NotFound desc = could not find container \"68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855\": container with ID starting with 68112ff37f16ea22b681e78c1f22ebf0d195dafc235409bb43d2bc3a3e57f855 not found: ID does not exist"
Sep 15 16:58:27.331436 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.331415    2578 scope.go:117] "RemoveContainer" containerID="ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"
Sep 15 16:58:27.331664 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:58:27.331648    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e\": container with ID starting with ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e not found: ID does not exist" containerID="ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"
Sep 15 16:58:27.331717 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:27.331671    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e"} err="failed to get container status \"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e\": rpc error: code = NotFound desc = could not find container \"ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e\": container with ID starting with ea87b83104de8698f1b1259c75882c5ad15436b881035164711dc84bf547da4e not found: ID does not exist"
Sep 15 16:58:29.154510 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:29.154475    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" path="/var/lib/kubelet/pods/a93c0eb2-017d-4950-9d62-714f0ff6e4b3/volumes"
Sep 15 16:58:31.246543 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:31.246457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:58:45.197462 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:45.197430    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:58:45.198015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:45.197668    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container" containerID="cri-o://7fa6f490e851f8247b818a78eccc6a43d74102010adf62b5b54f178ddd40dd79" gracePeriod=30
Sep 15 16:58:45.198015 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:45.197769    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" containerID="cri-o://7d4a5ba63ae46535cf74b7b941c0294f51c0aacd684f166a1ebd903477086e9c" gracePeriod=30
Sep 15 16:58:45.390869 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:45.390825    2578 generic.go:356] "Generic (PLEG): container finished" podID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerID="7d4a5ba63ae46535cf74b7b941c0294f51c0aacd684f166a1ebd903477086e9c" exitCode=2
Sep 15 16:58:45.391028 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:45.390901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerDied","Data":"7d4a5ba63ae46535cf74b7b941c0294f51c0aacd684f166a1ebd903477086e9c"}
Sep 15 16:58:46.241114 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:46.241073    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:58:50.385579 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 16:58:50.385983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385946    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="storage-initializer"
Sep 15 16:58:50.385983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385958    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="storage-initializer"
Sep 15 16:58:50.385983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385971    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kube-rbac-proxy"
Sep 15 16:58:50.385983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385977    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kube-rbac-proxy"
Sep 15 16:58:50.385983 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385985    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container"
Sep 15 16:58:50.386157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.385991    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container"
Sep 15 16:58:50.386157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.386056    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kserve-container"
Sep 15 16:58:50.386157 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.386064    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93c0eb2-017d-4950-9d62-714f0ff6e4b3" containerName="kube-rbac-proxy"
Sep 15 16:58:50.388651 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.388632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.391224 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.391193    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 15 16:58:50.391318 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.391293    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 15 16:58:50.404931 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.404903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 16:58:50.483776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.483726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpnz\" (UniqueName: \"kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.483776 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.483776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.483973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.483807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.483973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.483827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.483973 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.483912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585074 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585074 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpnz\" (UniqueName: \"kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585312 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585451 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:58:50.585328    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 15 16:58:50.585451 ip-10-0-141-27 kubenswrapper[2578]: E0915 16:58:50.585441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls podName:4ed73c57-9c82-4d4f-94a0-55b178f4c4ab nodeName:}" failed. No retries permitted until 2026-09-15 16:58:51.085411604 +0000 UTC m=+1902.472050858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls") pod "isvc-triton-predictor-5d7db58676-bvm7m" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab") : secret "isvc-triton-predictor-serving-cert" not found
Sep 15 16:58:50.585544 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585849 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.585893 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.585858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:50.593863 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:50.593837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpnz\" (UniqueName: \"kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:51.090659 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:51.090621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:51.093199 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:51.093178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bvm7m\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:51.241043 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:51.241004    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:58:51.299392 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:51.299363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 16:58:51.571902 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:51.571877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 16:58:51.574253 ip-10-0-141-27 kubenswrapper[2578]: W0915 16:58:51.574218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed73c57_9c82_4d4f_94a0_55b178f4c4ab.slice/crio-ad195ad9a04b86c8d5c9030a056ce1adf8f33fc6ffa7af85d1ab5700a6d29889 WatchSource:0}: Error finding container ad195ad9a04b86c8d5c9030a056ce1adf8f33fc6ffa7af85d1ab5700a6d29889: Status 404 returned error can't find the container with id ad195ad9a04b86c8d5c9030a056ce1adf8f33fc6ffa7af85d1ab5700a6d29889
Sep 15 16:58:52.426851 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:52.426814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerStarted","Data":"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"}
Sep 15 16:58:52.427038 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:52.426868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerStarted","Data":"ad195ad9a04b86c8d5c9030a056ce1adf8f33fc6ffa7af85d1ab5700a6d29889"}
Sep 15 16:58:55.442446 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:55.442420    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerID="75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe" exitCode=0
Sep 15 16:58:55.442855 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:55.442494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerDied","Data":"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"}
Sep 15 16:58:56.240667 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:56.240626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:58:56.240856 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:58:56.240800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:59:01.241795 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:01.241302    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:59:06.241128 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:06.241081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:59:11.240974 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:11.240854    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 15 16:59:15.608728 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:15.608631    2578 generic.go:356] "Generic (PLEG): container finished" podID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerID="7fa6f490e851f8247b818a78eccc6a43d74102010adf62b5b54f178ddd40dd79" exitCode=137
Sep 15 16:59:15.608728 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:15.608726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerDied","Data":"7fa6f490e851f8247b818a78eccc6a43d74102010adf62b5b54f178ddd40dd79"}
Sep 15 16:59:15.912427 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:15.912396    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:59:16.044835 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.044800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert\") pod \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") "
Sep 15 16:59:16.045047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.044853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls\") pod \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") "
Sep 15 16:59:16.045047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.044879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config\") pod \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") "
Sep 15 16:59:16.045047 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.044915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location\") pod \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") "
Sep 15 16:59:16.045241 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.045086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgnq\" (UniqueName: \"kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq\") pod \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\" (UID: \"066a15f1-8bd5-4d5d-a14a-8c95625d6781\") "
Sep 15 16:59:16.045479 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.045439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "066a15f1-8bd5-4d5d-a14a-8c95625d6781" (UID: "066a15f1-8bd5-4d5d-a14a-8c95625d6781"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:59:16.045825 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.045694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "066a15f1-8bd5-4d5d-a14a-8c95625d6781" (UID: "066a15f1-8bd5-4d5d-a14a-8c95625d6781"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:59:16.047998 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.047963    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq" (OuterVolumeSpecName: "kube-api-access-8wgnq") pod "066a15f1-8bd5-4d5d-a14a-8c95625d6781" (UID: "066a15f1-8bd5-4d5d-a14a-8c95625d6781"). InnerVolumeSpecName "kube-api-access-8wgnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:59:16.049194 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.049167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "066a15f1-8bd5-4d5d-a14a-8c95625d6781" (UID: "066a15f1-8bd5-4d5d-a14a-8c95625d6781"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:59:16.060257 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.060220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "066a15f1-8bd5-4d5d-a14a-8c95625d6781" (UID: "066a15f1-8bd5-4d5d-a14a-8c95625d6781"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:59:16.146557 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.146517    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:59:16.146557 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.146555    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066a15f1-8bd5-4d5d-a14a-8c95625d6781-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:59:16.147018 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.146565    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:59:16.147018 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.146574    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:59:16.147018 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.146583    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgnq\" (UniqueName: \"kubernetes.io/projected/066a15f1-8bd5-4d5d-a14a-8c95625d6781-kube-api-access-8wgnq\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 16:59:16.617517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.617423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs" event={"ID":"066a15f1-8bd5-4d5d-a14a-8c95625d6781","Type":"ContainerDied","Data":"7613d9e85be56a35b43d584726443860594fd9f62c5c7c73a610792196cd4f38"}
Sep 15 16:59:16.617517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.617487    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"
Sep 15 16:59:16.617517 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.617514    2578 scope.go:117] "RemoveContainer" containerID="7d4a5ba63ae46535cf74b7b941c0294f51c0aacd684f166a1ebd903477086e9c"
Sep 15 16:59:16.630353 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.630333    2578 scope.go:117] "RemoveContainer" containerID="7fa6f490e851f8247b818a78eccc6a43d74102010adf62b5b54f178ddd40dd79"
Sep 15 16:59:16.640414 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.640394    2578 scope.go:117] "RemoveContainer" containerID="711af1ab627a5385854a379b940fd660d562442f3732b749dd46bac512a6dbf4"
Sep 15 16:59:16.645887 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.645817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:59:16.648027 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:16.647986    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-jn5gs"]
Sep 15 16:59:17.157620 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:17.157585    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" path="/var/lib/kubelet/pods/066a15f1-8bd5-4d5d-a14a-8c95625d6781/volumes"
Sep 15 16:59:59.723702 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:59.723662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 16:59:59.724285 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:59.723962    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" containerID="cri-o://2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc" gracePeriod=30
Sep 15 16:59:59.724285 ip-10-0-141-27 kubenswrapper[2578]: I0915 16:59:59.724142    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy" containerID="cri-o://013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a" gracePeriod=30
Sep 15 17:00:01.892057 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:01.891962    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 15 17:00:01.898151 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:01.898115    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 15 17:00:04.942299 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942258    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942658    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container"
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942670    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container"
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="storage-initializer"
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942697    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="storage-initializer"
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942715    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy"
Sep 15 17:00:04.942777 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942721    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy"
Sep 15 17:00:04.943122 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942824    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kube-rbac-proxy"
Sep 15 17:00:04.943122 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.942844    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="066a15f1-8bd5-4d5d-a14a-8c95625d6781" containerName="kserve-container"
Sep 15 17:00:04.965255 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.965225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:00:04.965403 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.965301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:04.967868 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.967844    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 15 17:00:04.968046 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:04.967844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 15 17:00:05.034901 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.034867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.035089 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.034944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.035089 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.034987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.035089 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.035064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szqx\" (UniqueName: \"kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.035230 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.035094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.135876 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.135829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136107 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136107 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136238 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szqx\" (UniqueName: \"kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136238 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136342 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136994 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.136994 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.136920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.139137 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.139112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.143866 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.143797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szqx\" (UniqueName: \"kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-bms8l\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:05.277234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:05.277197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:06.891835 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:06.891787    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 15 17:00:08.904049 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.903983    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 17:00:08.929808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929778    2578 generic.go:356] "Generic (PLEG): container finished" podID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerID="013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a" exitCode=2
Sep 15 17:00:08.929808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929804    2578 generic.go:356] "Generic (PLEG): container finished" podID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerID="2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc" exitCode=0
Sep 15 17:00:08.930010 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929852    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"
Sep 15 17:00:08.930010 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerDied","Data":"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"}
Sep 15 17:00:08.930010 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerDied","Data":"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"}
Sep 15 17:00:08.930010 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx" event={"ID":"5459b0bc-4086-4a08-b186-6194fa6dec9e","Type":"ContainerDied","Data":"33816ba9f3dd98b7240d9b31dfc383cf811b61a02bca73436e9c8b83d7150087"}
Sep 15 17:00:08.930010 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.929995    2578 scope.go:117] "RemoveContainer" containerID="013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"
Sep 15 17:00:08.974309 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhcmd\" (UniqueName: \"kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd\") pod \"5459b0bc-4086-4a08-b186-6194fa6dec9e\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") "
Sep 15 17:00:08.974445 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config\") pod \"5459b0bc-4086-4a08-b186-6194fa6dec9e\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") "
Sep 15 17:00:08.974445 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert\") pod \"5459b0bc-4086-4a08-b186-6194fa6dec9e\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") "
Sep 15 17:00:08.974445 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location\") pod \"5459b0bc-4086-4a08-b186-6194fa6dec9e\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") "
Sep 15 17:00:08.974575 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") pod \"5459b0bc-4086-4a08-b186-6194fa6dec9e\" (UID: \"5459b0bc-4086-4a08-b186-6194fa6dec9e\") "
Sep 15 17:00:08.974846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5459b0bc-4086-4a08-b186-6194fa6dec9e" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:00:08.974846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5459b0bc-4086-4a08-b186-6194fa6dec9e" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:08.974846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.974828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5459b0bc-4086-4a08-b186-6194fa6dec9e" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:08.976457 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.976434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd" (OuterVolumeSpecName: "kube-api-access-fhcmd") pod "5459b0bc-4086-4a08-b186-6194fa6dec9e" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e"). InnerVolumeSpecName "kube-api-access-fhcmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:08.976524 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.976475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5459b0bc-4086-4a08-b186-6194fa6dec9e" (UID: "5459b0bc-4086-4a08-b186-6194fa6dec9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:08.991181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:08.991160    2578 scope.go:117] "RemoveContainer" containerID="2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"
Sep 15 17:00:09.075853 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.075822    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5459b0bc-4086-4a08-b186-6194fa6dec9e-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:00:09.075853 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.075855    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5459b0bc-4086-4a08-b186-6194fa6dec9e-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:00:09.076072 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.075872    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcmd\" (UniqueName: \"kubernetes.io/projected/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-api-access-fhcmd\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:00:09.076072 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.075885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:00:09.076072 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.075896    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5459b0bc-4086-4a08-b186-6194fa6dec9e-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:00:09.248950 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.248921    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 17:00:09.253067 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.253039    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-hl4wx"]
Sep 15 17:00:09.262563 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.262534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:00:09.824435 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:00:09.824339    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ef1660_fc22_4cad_a043_847f0f29a040.slice/crio-522be8c8a7039843719269faa73139e566fefd39fb5316610035cd071474e1e6 WatchSource:0}: Error finding container 522be8c8a7039843719269faa73139e566fefd39fb5316610035cd071474e1e6: Status 404 returned error can't find the container with id 522be8c8a7039843719269faa73139e566fefd39fb5316610035cd071474e1e6
Sep 15 17:00:09.825757 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.825720    2578 scope.go:117] "RemoveContainer" containerID="3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"
Sep 15 17:00:09.937257 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:09.937221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerStarted","Data":"522be8c8a7039843719269faa73139e566fefd39fb5316610035cd071474e1e6"}
Sep 15 17:00:10.957014 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.956992    2578 scope.go:117] "RemoveContainer" containerID="013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"
Sep 15 17:00:10.957347 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:00:10.957302    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a\": container with ID starting with 013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a not found: ID does not exist" containerID="013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"
Sep 15 17:00:10.957347 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.957330    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"} err="failed to get container status \"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a\": rpc error: code = NotFound desc = could not find container \"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a\": container with ID starting with 013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a not found: ID does not exist"
Sep 15 17:00:10.957453 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.957348    2578 scope.go:117] "RemoveContainer" containerID="2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"
Sep 15 17:00:10.957653 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:00:10.957614    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc\": container with ID starting with 2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc not found: ID does not exist" containerID="2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"
Sep 15 17:00:10.957699 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.957664    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"} err="failed to get container status \"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc\": rpc error: code = NotFound desc = could not find container \"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc\": container with ID starting with 2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc not found: ID does not exist"
Sep 15 17:00:10.957699 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.957680    2578 scope.go:117] "RemoveContainer" containerID="3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"
Sep 15 17:00:10.958083 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:00:10.958065    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566\": container with ID starting with 3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566 not found: ID does not exist" containerID="3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"
Sep 15 17:00:10.958161 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958085    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"} err="failed to get container status \"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566\": rpc error: code = NotFound desc = could not find container \"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566\": container with ID starting with 3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566 not found: ID does not exist"
Sep 15 17:00:10.958161 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958100    2578 scope.go:117] "RemoveContainer" containerID="013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"
Sep 15 17:00:10.958336 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958314    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a"} err="failed to get container status \"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a\": rpc error: code = NotFound desc = could not find container \"013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a\": container with ID starting with 013ef5408148910e7e37cd587204b1402c27773f7789bd3ed65e272849c7f84a not found: ID does not exist"
Sep 15 17:00:10.958379 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958337    2578 scope.go:117] "RemoveContainer" containerID="2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"
Sep 15 17:00:10.958555 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958536    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc"} err="failed to get container status \"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc\": rpc error: code = NotFound desc = could not find container \"2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc\": container with ID starting with 2d74f12045cc131b2b9bf87b0bb26616bd9b189cdafb8ba16ad72d3925492ddc not found: ID does not exist"
Sep 15 17:00:10.958609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958555    2578 scope.go:117] "RemoveContainer" containerID="3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"
Sep 15 17:00:10.958799 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:10.958781    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566"} err="failed to get container status \"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566\": rpc error: code = NotFound desc = could not find container \"3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566\": container with ID starting with 3d3e5ac530f5f064174400ff335ef346b8a72ad9b10d2dc9182d1787fb9d3566 not found: ID does not exist"
Sep 15 17:00:11.156295 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:11.156256    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" path="/var/lib/kubelet/pods/5459b0bc-4086-4a08-b186-6194fa6dec9e/volumes"
Sep 15 17:00:20.997425 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:20.997328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerStarted","Data":"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"}
Sep 15 17:00:50.149795 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:50.149759    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerID="7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003" exitCode=0
Sep 15 17:00:50.150194 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:50.149821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerDied","Data":"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"}
Sep 15 17:00:51.161966 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.161933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerStarted","Data":"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"}
Sep 15 17:00:51.162475 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.161998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerStarted","Data":"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"}
Sep 15 17:00:51.162475 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.162036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:00:51.162475 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.162070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:00:51.163293 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.163250    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 15 17:00:51.164234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.164211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerStarted","Data":"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"}
Sep 15 17:00:51.164318 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.164262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerStarted","Data":"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"}
Sep 15 17:00:51.164452 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.164439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:51.179975 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.179927    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podStartSLOduration=1.957560773 podStartE2EDuration="2m1.179908513s" totalImagesPullingTime="1m55.425092719s" totalInitContainerRuntime="3.797255021s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:51.179085216 +0000 UTC m=+2022.565724484" watchObservedRunningTime="2026-09-15 17:00:51.179908513 +0000 UTC m=+2022.566547781"
Sep 15 17:00:51.196358 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:51.196308    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podStartSLOduration=43.145805089 podStartE2EDuration="47.196290463s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.050485374s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:51.195167455 +0000 UTC m=+2022.581806722" watchObservedRunningTime="2026-09-15 17:00:51.196290463 +0000 UTC m=+2022.582929730"
Sep 15 17:00:52.168697 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:52.168665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:52.169146 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:52.168659    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 15 17:00:52.169774 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:52.169729    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:00:53.172538 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:53.172491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:00:57.172878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:57.172846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:00:57.173398 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:57.173361    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 15 17:00:58.176715 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:58.176688    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:00:58.177346 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:00:58.177322    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:07.174205 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:07.174176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:01:08.178155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:08.178121    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:14.301900 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:14.301866    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 17:01:14.302429 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:14.302112    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container" containerID="cri-o://a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773" gracePeriod=30
Sep 15 17:01:14.302429 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:14.302174    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kube-rbac-proxy" containerID="cri-o://f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a" gracePeriod=30
Sep 15 17:01:15.273896 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:15.273864    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerID="f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a" exitCode=2
Sep 15 17:01:15.274066 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:15.273908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerDied","Data":"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"}
Sep 15 17:01:17.180048 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.180024    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:01:17.286306 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.286184    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerID="a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773" exitCode=0
Sep 15 17:01:17.286480 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.286307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"
Sep 15 17:01:17.286892 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.286311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerDied","Data":"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"}
Sep 15 17:01:17.287001 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.286981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m" event={"ID":"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab","Type":"ContainerDied","Data":"ad195ad9a04b86c8d5c9030a056ce1adf8f33fc6ffa7af85d1ab5700a6d29889"}
Sep 15 17:01:17.287096 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.287039    2578 scope.go:117] "RemoveContainer" containerID="f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"
Sep 15 17:01:17.299870 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.299840    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfpnz\" (UniqueName: \"kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz\") pod \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") "
Sep 15 17:01:17.300000 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.299912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert\") pod \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") "
Sep 15 17:01:17.300000 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.299980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") pod \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") "
Sep 15 17:01:17.300177 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.300032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config\") pod \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") "
Sep 15 17:01:17.300177 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.300068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location\") pod \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\" (UID: \"4ed73c57-9c82-4d4f-94a0-55b178f4c4ab\") "
Sep 15 17:01:17.300604 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.300326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:17.300604 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.300506    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:17.301264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.301225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:17.302045 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.302019    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:01:17.302236 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.302209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz" (OuterVolumeSpecName: "kube-api-access-qfpnz") pod "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab"). InnerVolumeSpecName "kube-api-access-qfpnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:01:17.302410 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.302392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" (UID: "4ed73c57-9c82-4d4f-94a0-55b178f4c4ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:01:17.304507 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.304488    2578 scope.go:117] "RemoveContainer" containerID="a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"
Sep 15 17:01:17.318029 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.318008    2578 scope.go:117] "RemoveContainer" containerID="75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"
Sep 15 17:01:17.326103 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.326086    2578 scope.go:117] "RemoveContainer" containerID="f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"
Sep 15 17:01:17.326391 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:17.326371    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a\": container with ID starting with f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a not found: ID does not exist" containerID="f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"
Sep 15 17:01:17.326468 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.326405    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a"} err="failed to get container status \"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a\": rpc error: code = NotFound desc = could not find container \"f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a\": container with ID starting with f483da22d687fd0c5ff6b71c77bdd0f2336748f993cc838e72d04c832bcb695a not found: ID does not exist"
Sep 15 17:01:17.326468 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.326430    2578 scope.go:117] "RemoveContainer" containerID="a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"
Sep 15 17:01:17.326668 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:17.326651    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773\": container with ID starting with a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773 not found: ID does not exist" containerID="a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"
Sep 15 17:01:17.326759 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.326676    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773"} err="failed to get container status \"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773\": rpc error: code = NotFound desc = could not find container \"a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773\": container with ID starting with a25d5bfc79a81189e1fde3bb0775f61d9a29306f2ac9c2ea842473c192d77773 not found: ID does not exist"
Sep 15 17:01:17.326759 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.326696    2578 scope.go:117] "RemoveContainer" containerID="75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"
Sep 15 17:01:17.327032 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:17.327011    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe\": container with ID starting with 75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe not found: ID does not exist" containerID="75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"
Sep 15 17:01:17.327088 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.327036    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe"} err="failed to get container status \"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe\": rpc error: code = NotFound desc = could not find container \"75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe\": container with ID starting with 75819a9ff061613d4c464ba49e775c79aa22101efa90d48b68defb317653adfe not found: ID does not exist"
Sep 15 17:01:17.401673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.401634    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:17.401673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.401669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:17.401673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.401678    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:17.401911 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.401687    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpnz\" (UniqueName: \"kubernetes.io/projected/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab-kube-api-access-qfpnz\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:17.609002 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.608972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 17:01:17.612748 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:17.612710    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bvm7m"]
Sep 15 17:01:18.177759 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:18.177698    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:19.154323 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.154287    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" path="/var/lib/kubelet/pods/4ed73c57-9c82-4d4f-94a0-55b178f4c4ab/volumes"
Sep 15 17:01:19.491318 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491229    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:01:19.491841 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="storage-initializer"
Sep 15 17:01:19.491890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="storage-initializer"
Sep 15 17:01:19.491890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491864    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kube-rbac-proxy"
Sep 15 17:01:19.491890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kube-rbac-proxy"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491889    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491912    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="storage-initializer"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491920    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="storage-initializer"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491933    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491941    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491961    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container"
Sep 15 17:01:19.491991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.491971    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container"
Sep 15 17:01:19.492234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.492069    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kube-rbac-proxy"
Sep 15 17:01:19.492234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.492082    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kserve-container"
Sep 15 17:01:19.492234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.492093    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5459b0bc-4086-4a08-b186-6194fa6dec9e" containerName="kube-rbac-proxy"
Sep 15 17:01:19.492234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.492101    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed73c57-9c82-4d4f-94a0-55b178f4c4ab" containerName="kserve-container"
Sep 15 17:01:19.518945 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.518912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:01:19.519119 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.519048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.521403 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.521373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 15 17:01:19.521527 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.521408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 15 17:01:19.620084 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.620050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.620264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.620098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.620264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.620129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.620264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.620145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.620264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.620164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258dv\" (UniqueName: \"kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.720878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.720848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721098 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.720891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721098 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.720921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721098 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.721033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721098 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.721066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258dv\" (UniqueName: \"kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721334 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.721260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721608 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.721587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.721683 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.721605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.723618 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.723594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.729287 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.729265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258dv\" (UniqueName: \"kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv\") pod \"isvc-xgboost-predictor-7566d65bf7-p2t9k\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:19.829879 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:19.829795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:20.121111 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:20.121084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:01:20.123610 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:01:20.123581    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe53e1f_0d49_418a_a0fc_c05538755a09.slice/crio-6de19bdd099f425501a8774716d014879272c142b65648caa23e2787c6eb8237 WatchSource:0}: Error finding container 6de19bdd099f425501a8774716d014879272c142b65648caa23e2787c6eb8237: Status 404 returned error can't find the container with id 6de19bdd099f425501a8774716d014879272c142b65648caa23e2787c6eb8237
Sep 15 17:01:20.305945 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:20.305906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerStarted","Data":"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"}
Sep 15 17:01:20.306441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:20.305967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerStarted","Data":"6de19bdd099f425501a8774716d014879272c142b65648caa23e2787c6eb8237"}
Sep 15 17:01:24.327305 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:24.327273    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerID="473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2" exitCode=0
Sep 15 17:01:24.327686 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:24.327336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerDied","Data":"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"}
Sep 15 17:01:28.177919 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:28.177880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:28.489407 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:28.489316    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:01:28.489689 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:28.489635    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" containerID="cri-o://e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca" gracePeriod=30
Sep 15 17:01:28.489871 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:28.489693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" containerID="cri-o://39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2" gracePeriod=30
Sep 15 17:01:28.489960 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:28.489882    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:29.360321 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:29.360283    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerID="39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2" exitCode=2
Sep 15 17:01:29.360810 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:29.360367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerDied","Data":"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"}
Sep 15 17:01:33.172868 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:33.172762    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:34.988623 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:34.988584    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:01:34.991502 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:34.991482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.000703 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.000679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:01:35.071568 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.071529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.071773 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.071582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.071773 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.071611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.071773 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.071672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dj6h\" (UniqueName: \"kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.071908 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.071777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172678 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172678 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dj6h\" (UniqueName: \"kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172678 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.172936 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.172907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.173275 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.173247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.173398 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.173259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.175213 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.175192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.180910 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.180887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dj6h\" (UniqueName: \"kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-ntzm8\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:35.304983 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:35.304903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:38.173718 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:38.173666    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:38.490904 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:38.490794    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:43.173450 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:43.173388    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:43.173928 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:43.173566    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:01:43.442985 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:43.442957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerStarted","Data":"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"}
Sep 15 17:01:43.514359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:43.514165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:01:43.517780 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:01:43.517389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc852f3_7157_4499_ad1c_f88e6bff3e02.slice/crio-a6f7523aa73399ab702e62ea73149d4b3345240c817cb9adb00a968bd8f56515 WatchSource:0}: Error finding container a6f7523aa73399ab702e62ea73149d4b3345240c817cb9adb00a968bd8f56515: Status 404 returned error can't find the container with id a6f7523aa73399ab702e62ea73149d4b3345240c817cb9adb00a968bd8f56515
Sep 15 17:01:44.449519 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.449479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerStarted","Data":"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"}
Sep 15 17:01:44.450024 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.449557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerStarted","Data":"a6f7523aa73399ab702e62ea73149d4b3345240c817cb9adb00a968bd8f56515"}
Sep 15 17:01:44.451598 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.451573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerStarted","Data":"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"}
Sep 15 17:01:44.451754 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.451718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:44.451830 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.451780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:44.453367 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.453343    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:01:44.487372 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:44.487313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podStartSLOduration=2.5698182000000003 podStartE2EDuration="25.487296792s" totalImagesPullingTime="19.022688314s" totalInitContainerRuntime="3.894790278s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:44.485621505 +0000 UTC m=+2075.872260773" watchObservedRunningTime="2026-09-15 17:01:44.487296792 +0000 UTC m=+2075.873936059"
Sep 15 17:01:45.457154 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:45.457114    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:01:48.173098 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:48.173059    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:48.475065 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:48.474981    2578 generic.go:356] "Generic (PLEG): container finished" podID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerID="a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297" exitCode=0
Sep 15 17:01:48.475065 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:48.475055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerDied","Data":"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"}
Sep 15 17:01:48.490722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:48.490689    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:49.482704 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:49.482670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerStarted","Data":"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"}
Sep 15 17:01:49.483201 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:49.482764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerStarted","Data":"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"}
Sep 15 17:01:49.483201 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:49.482985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:49.500877 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:49.500819    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podStartSLOduration=11.504267285 podStartE2EDuration="15.500800319s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996533034s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:49.499901548 +0000 UTC m=+2080.886540816" watchObservedRunningTime="2026-09-15 17:01:49.500800319 +0000 UTC m=+2080.887439583"
Sep 15 17:01:50.462515 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:50.462486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:01:50.463042 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:50.463017    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:01:50.488212 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:50.488182    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:50.489459 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:50.489431    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:01:51.495158 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:51.495110    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:01:53.173107 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:53.173065    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:56.499264 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:56.499235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:01:56.499760 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:56.499708    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:01:58.172870 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:58.172828    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 15 17:01:58.490210 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:58.490106    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 15 17:01:59.312595 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.312569    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:01:59.397984 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.397947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls\") pod \"f5ef1660-fc22-4cad-a043-847f0f29a040\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") "
Sep 15 17:01:59.398175 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert\") pod \"f5ef1660-fc22-4cad-a043-847f0f29a040\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") "
Sep 15 17:01:59.398175 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location\") pod \"f5ef1660-fc22-4cad-a043-847f0f29a040\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") "
Sep 15 17:01:59.398305 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config\") pod \"f5ef1660-fc22-4cad-a043-847f0f29a040\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") "
Sep 15 17:01:59.398305 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2szqx\" (UniqueName: \"kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx\") pod \"f5ef1660-fc22-4cad-a043-847f0f29a040\" (UID: \"f5ef1660-fc22-4cad-a043-847f0f29a040\") "
Sep 15 17:01:59.398511 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5ef1660-fc22-4cad-a043-847f0f29a040" (UID: "f5ef1660-fc22-4cad-a043-847f0f29a040"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:59.398554 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398498    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5ef1660-fc22-4cad-a043-847f0f29a040" (UID: "f5ef1660-fc22-4cad-a043-847f0f29a040"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:01:59.398554 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.398520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5ef1660-fc22-4cad-a043-847f0f29a040" (UID: "f5ef1660-fc22-4cad-a043-847f0f29a040"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:59.400118 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.400097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5ef1660-fc22-4cad-a043-847f0f29a040" (UID: "f5ef1660-fc22-4cad-a043-847f0f29a040"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:01:59.400386 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.400369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx" (OuterVolumeSpecName: "kube-api-access-2szqx") pod "f5ef1660-fc22-4cad-a043-847f0f29a040" (UID: "f5ef1660-fc22-4cad-a043-847f0f29a040"). InnerVolumeSpecName "kube-api-access-2szqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:01:59.498982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.498888    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:59.498982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.498918    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ef1660-fc22-4cad-a043-847f0f29a040-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:59.498982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.498929    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:59.498982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.498944    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2szqx\" (UniqueName: \"kubernetes.io/projected/f5ef1660-fc22-4cad-a043-847f0f29a040-kube-api-access-2szqx\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:59.498982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.498956    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ef1660-fc22-4cad-a043-847f0f29a040-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:01:59.534081 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.534047    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerID="e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca" exitCode=137
Sep 15 17:01:59.534243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.534106    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"
Sep 15 17:01:59.534243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.534107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerDied","Data":"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"}
Sep 15 17:01:59.534243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.534238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l" event={"ID":"f5ef1660-fc22-4cad-a043-847f0f29a040","Type":"ContainerDied","Data":"522be8c8a7039843719269faa73139e566fefd39fb5316610035cd071474e1e6"}
Sep 15 17:01:59.534353 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.534269    2578 scope.go:117] "RemoveContainer" containerID="39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"
Sep 15 17:01:59.543970 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.543953    2578 scope.go:117] "RemoveContainer" containerID="e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"
Sep 15 17:01:59.551472 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.551453    2578 scope.go:117] "RemoveContainer" containerID="7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"
Sep 15 17:01:59.556602 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.556574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:01:59.560440 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.560420    2578 scope.go:117] "RemoveContainer" containerID="39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"
Sep 15 17:01:59.560585 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.560568    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-bms8l"]
Sep 15 17:01:59.560725 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:59.560707    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2\": container with ID starting with 39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2 not found: ID does not exist" containerID="39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"
Sep 15 17:01:59.560783 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.560758    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2"} err="failed to get container status \"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2\": rpc error: code = NotFound desc = could not find container \"39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2\": container with ID starting with 39e74fbe66c76fa3d75f5aa70c3c199e9d2d43af2db38abfa11b92ce621382b2 not found: ID does not exist"
Sep 15 17:01:59.560783 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.560777    2578 scope.go:117] "RemoveContainer" containerID="e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"
Sep 15 17:01:59.561047 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:59.561030    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca\": container with ID starting with e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca not found: ID does not exist" containerID="e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"
Sep 15 17:01:59.561117 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.561056    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca"} err="failed to get container status \"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca\": rpc error: code = NotFound desc = could not find container \"e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca\": container with ID starting with e8b5a2fdc70f7ded5b87e8f3aba89bbf23e5bdc50f8decb73e3bf005706883ca not found: ID does not exist"
Sep 15 17:01:59.561117 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.561079    2578 scope.go:117] "RemoveContainer" containerID="7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"
Sep 15 17:01:59.561363 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:01:59.561346    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003\": container with ID starting with 7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003 not found: ID does not exist" containerID="7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"
Sep 15 17:01:59.561412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:01:59.561368    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003"} err="failed to get container status \"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003\": rpc error: code = NotFound desc = could not find container \"7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003\": container with ID starting with 7a0d6d280e69c72b4dcdf0b38f696afe82ec0465f5a2c6b5ca6cccedecead003 not found: ID does not exist"
Sep 15 17:02:00.463501 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:00.463464    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:02:01.154115 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:01.154084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" path="/var/lib/kubelet/pods/f5ef1660-fc22-4cad-a043-847f0f29a040/volumes"
Sep 15 17:02:06.500500 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:06.500459    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:10.463951 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:10.463909    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:02:16.499665 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:16.499621    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:20.463633 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:20.463589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:02:26.500470 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:26.500426    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:30.463133 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:30.463091    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:02:36.500366 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:36.500324    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:40.463525 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:40.463488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:02:46.499981 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:46.499938    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:50.464439 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:50.464412    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:02:56.500518 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:56.500478    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:02:59.587539 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:59.587499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:02:59.587986 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:59.587856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" containerID="cri-o://6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12" gracePeriod=30
Sep 15 17:02:59.587986 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:59.587878    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kube-rbac-proxy" containerID="cri-o://923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb" gracePeriod=30
Sep 15 17:02:59.817858 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:59.817826    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerID="923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb" exitCode=2
Sep 15 17:02:59.818017 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:02:59.817904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerDied","Data":"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"}
Sep 15 17:03:00.457992 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:00.457945    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 15 17:03:00.463692 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:00.463666    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 15 17:03:03.230074 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.230047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:03:03.251759 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.251668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert\") pod \"bbe53e1f-0d49-418a-a0fc-c05538755a09\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") "
Sep 15 17:03:03.251901 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.251782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls\") pod \"bbe53e1f-0d49-418a-a0fc-c05538755a09\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") "
Sep 15 17:03:03.251901 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.251826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-258dv\" (UniqueName: \"kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv\") pod \"bbe53e1f-0d49-418a-a0fc-c05538755a09\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") "
Sep 15 17:03:03.251901 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.251871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config\") pod \"bbe53e1f-0d49-418a-a0fc-c05538755a09\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") "
Sep 15 17:03:03.252074 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.251929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location\") pod \"bbe53e1f-0d49-418a-a0fc-c05538755a09\" (UID: \"bbe53e1f-0d49-418a-a0fc-c05538755a09\") "
Sep 15 17:03:03.252137 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.252108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbe53e1f-0d49-418a-a0fc-c05538755a09" (UID: "bbe53e1f-0d49-418a-a0fc-c05538755a09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:03.252268 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.252251    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:03.252388 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.252355    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbe53e1f-0d49-418a-a0fc-c05538755a09" (UID: "bbe53e1f-0d49-418a-a0fc-c05538755a09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:03.252492 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.252386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbe53e1f-0d49-418a-a0fc-c05538755a09" (UID: "bbe53e1f-0d49-418a-a0fc-c05538755a09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:03:03.254552 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.254532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbe53e1f-0d49-418a-a0fc-c05538755a09" (UID: "bbe53e1f-0d49-418a-a0fc-c05538755a09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:03:03.254632 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.254556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv" (OuterVolumeSpecName: "kube-api-access-258dv") pod "bbe53e1f-0d49-418a-a0fc-c05538755a09" (UID: "bbe53e1f-0d49-418a-a0fc-c05538755a09"). InnerVolumeSpecName "kube-api-access-258dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:03:03.352983 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.352947    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbe53e1f-0d49-418a-a0fc-c05538755a09-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:03.352983 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.352977    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258dv\" (UniqueName: \"kubernetes.io/projected/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-api-access-258dv\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:03.352983 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.352987    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbe53e1f-0d49-418a-a0fc-c05538755a09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:03.353217 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.352998    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbe53e1f-0d49-418a-a0fc-c05538755a09-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:03.837382 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.837346    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerID="6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12" exitCode=0
Sep 15 17:03:03.837542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.837398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerDied","Data":"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"}
Sep 15 17:03:03.837542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.837417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"
Sep 15 17:03:03.837542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.837461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k" event={"ID":"bbe53e1f-0d49-418a-a0fc-c05538755a09","Type":"ContainerDied","Data":"6de19bdd099f425501a8774716d014879272c142b65648caa23e2787c6eb8237"}
Sep 15 17:03:03.837542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.837499    2578 scope.go:117] "RemoveContainer" containerID="923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"
Sep 15 17:03:03.846827 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.846809    2578 scope.go:117] "RemoveContainer" containerID="6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"
Sep 15 17:03:03.854447 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.854427    2578 scope.go:117] "RemoveContainer" containerID="473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"
Sep 15 17:03:03.860110 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.860084    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:03:03.862369 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.862355    2578 scope.go:117] "RemoveContainer" containerID="923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"
Sep 15 17:03:03.863329 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:03.863043    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb\": container with ID starting with 923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb not found: ID does not exist" containerID="923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"
Sep 15 17:03:03.863329 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.863139    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb"} err="failed to get container status \"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb\": rpc error: code = NotFound desc = could not find container \"923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb\": container with ID starting with 923784b729b7e5f3791e8b4abe598d5b0b49b3663010051fdd7e8a976136bceb not found: ID does not exist"
Sep 15 17:03:03.863329 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.863168    2578 scope.go:117] "RemoveContainer" containerID="6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"
Sep 15 17:03:03.868876 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:03.863615    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12\": container with ID starting with 6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12 not found: ID does not exist" containerID="6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"
Sep 15 17:03:03.868876 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.864067    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12"} err="failed to get container status \"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12\": rpc error: code = NotFound desc = could not find container \"6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12\": container with ID starting with 6a16733cc2b304e7855d4e4fe00b9a36b78dad1bd7540a93a496d7fca870aa12 not found: ID does not exist"
Sep 15 17:03:03.868876 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.864100    2578 scope.go:117] "RemoveContainer" containerID="473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"
Sep 15 17:03:03.869262 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:03.869235    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2\": container with ID starting with 473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2 not found: ID does not exist" containerID="473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"
Sep 15 17:03:03.869404 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.869370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2"} err="failed to get container status \"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2\": rpc error: code = NotFound desc = could not find container \"473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2\": container with ID starting with 473b5cbf7c5a0656bec800ec45d2a66e50bd5d9346ced59eb03b70ad173768f2 not found: ID does not exist"
Sep 15 17:03:03.872347 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:03.872326    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-p2t9k"]
Sep 15 17:03:04.767624 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.767590    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.767987    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container"
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container"
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768008    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kube-rbac-proxy"
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768014    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kube-rbac-proxy"
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768027    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container"
Sep 15 17:03:04.768039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768033    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768043    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="storage-initializer"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768049    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="storage-initializer"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768059    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="storage-initializer"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768064    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="storage-initializer"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768070    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768075    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kserve-container"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768135    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kserve-container"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768141    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" containerName="kube-rbac-proxy"
Sep 15 17:03:04.768280 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.768146    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ef1660-fc22-4cad-a043-847f0f29a040" containerName="kube-rbac-proxy"
Sep 15 17:03:04.771637 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.771616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.773860 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.773836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 15 17:03:04.773958 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.773906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 15 17:03:04.780344 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.780319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:03:04.865022 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.864995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.865191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.865069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.865191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.865127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.865191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.865161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.865191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.865185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxkh\" (UniqueName: \"kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxkh\" (UniqueName: \"kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966558 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966881 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.966937 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.966926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.968698 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.968675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:04.974359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:04.974341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxkh\" (UniqueName: \"kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:05.081791 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:05.081682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:05.155302 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:05.155233    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe53e1f-0d49-418a-a0fc-c05538755a09" path="/var/lib/kubelet/pods/bbe53e1f-0d49-418a-a0fc-c05538755a09/volumes"
Sep 15 17:03:05.353956 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:05.353925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:03:05.357687 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:03:05.357658    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc491107d_b693_4efd_a928_4a8a3532702e.slice/crio-9541f10b947848524ace9bbc7388ea8b4ccdad2f4cb03a611c745cf8752f3596 WatchSource:0}: Error finding container 9541f10b947848524ace9bbc7388ea8b4ccdad2f4cb03a611c745cf8752f3596: Status 404 returned error can't find the container with id 9541f10b947848524ace9bbc7388ea8b4ccdad2f4cb03a611c745cf8752f3596
Sep 15 17:03:05.850318 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:05.850280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerStarted","Data":"78d0025b09b000f8afe2d75ab3e5189337d7ec6082abae901ceb5e7dbe6365a1"}
Sep 15 17:03:05.850849 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:05.850334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerStarted","Data":"9541f10b947848524ace9bbc7388ea8b4ccdad2f4cb03a611c745cf8752f3596"}
Sep 15 17:03:06.500882 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:06.500854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:03:09.871115 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:09.871081    2578 generic.go:356] "Generic (PLEG): container finished" podID="c491107d-b693-4efd-a928-4a8a3532702e" containerID="78d0025b09b000f8afe2d75ab3e5189337d7ec6082abae901ceb5e7dbe6365a1" exitCode=0
Sep 15 17:03:09.871541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:09.871126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerDied","Data":"78d0025b09b000f8afe2d75ab3e5189337d7ec6082abae901ceb5e7dbe6365a1"}
Sep 15 17:03:10.877877 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:10.877843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerStarted","Data":"1e710b056715347a981796e22167f1bf19bc3205ac68bedbe2afcd81f41b44c9"}
Sep 15 17:03:10.878283 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:10.877907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerStarted","Data":"9fa75689aa167ebc1b2eaf56e9718616c817505e151b67a34bbf00abbead0495"}
Sep 15 17:03:10.878283 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:10.878089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:10.898081 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:10.898030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" podStartSLOduration=3.06195846 podStartE2EDuration="6.898013732s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.836055272s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:03:10.896580661 +0000 UTC m=+2162.283219954" watchObservedRunningTime="2026-09-15 17:03:10.898013732 +0000 UTC m=+2162.284652999"
Sep 15 17:03:11.882305 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:11.882272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:15.062021 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:15.061989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:03:15.062504 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:15.062205    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" containerID="cri-o://1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13" gracePeriod=30
Sep 15 17:03:15.062504 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:15.062255    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kube-rbac-proxy" containerID="cri-o://84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41" gracePeriod=30
Sep 15 17:03:15.903448 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:15.903413    2578 generic.go:356] "Generic (PLEG): container finished" podID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerID="84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41" exitCode=2
Sep 15 17:03:15.903625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:15.903483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerDied","Data":"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"}
Sep 15 17:03:16.495889 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:16.495846    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 15 17:03:16.500126 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:16.500101    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 15 17:03:17.893155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:17.893125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:19.806871 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.806849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:03:19.903394 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config\") pod \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") "
Sep 15 17:03:19.903567 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls\") pod \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") "
Sep 15 17:03:19.903567 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location\") pod \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") "
Sep 15 17:03:19.903567 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dj6h\" (UniqueName: \"kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h\") pod \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") "
Sep 15 17:03:19.903567 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert\") pod \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\" (UID: \"0cc852f3-7157-4499-ad1c-f88e6bff3e02\") "
Sep 15 17:03:19.903842 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0cc852f3-7157-4499-ad1c-f88e6bff3e02" (UID: "0cc852f3-7157-4499-ad1c-f88e6bff3e02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:19.903897 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0cc852f3-7157-4499-ad1c-f88e6bff3e02" (UID: "0cc852f3-7157-4499-ad1c-f88e6bff3e02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:03:19.903939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.903903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0cc852f3-7157-4499-ad1c-f88e6bff3e02" (UID: "0cc852f3-7157-4499-ad1c-f88e6bff3e02"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:19.905693 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.905668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0cc852f3-7157-4499-ad1c-f88e6bff3e02" (UID: "0cc852f3-7157-4499-ad1c-f88e6bff3e02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:03:19.905827 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.905707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h" (OuterVolumeSpecName: "kube-api-access-8dj6h") pod "0cc852f3-7157-4499-ad1c-f88e6bff3e02" (UID: "0cc852f3-7157-4499-ad1c-f88e6bff3e02"). InnerVolumeSpecName "kube-api-access-8dj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:03:19.922579 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.922551    2578 generic.go:356] "Generic (PLEG): container finished" podID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerID="1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13" exitCode=0
Sep 15 17:03:19.922711 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.922600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerDied","Data":"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"}
Sep 15 17:03:19.922711 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.922616    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"
Sep 15 17:03:19.922711 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.922650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8" event={"ID":"0cc852f3-7157-4499-ad1c-f88e6bff3e02","Type":"ContainerDied","Data":"a6f7523aa73399ab702e62ea73149d4b3345240c817cb9adb00a968bd8f56515"}
Sep 15 17:03:19.922711 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.922678    2578 scope.go:117] "RemoveContainer" containerID="84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"
Sep 15 17:03:19.931549 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.931529    2578 scope.go:117] "RemoveContainer" containerID="1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"
Sep 15 17:03:19.939278 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.939261    2578 scope.go:117] "RemoveContainer" containerID="a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"
Sep 15 17:03:19.943642 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.943621    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:03:19.947238 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.947219    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-ntzm8"]
Sep 15 17:03:19.948049 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948033    2578 scope.go:117] "RemoveContainer" containerID="84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"
Sep 15 17:03:19.948348 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:19.948332    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41\": container with ID starting with 84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41 not found: ID does not exist" containerID="84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"
Sep 15 17:03:19.948391 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948357    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41"} err="failed to get container status \"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41\": rpc error: code = NotFound desc = could not find container \"84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41\": container with ID starting with 84842887fad6e893a32247bc01d70660b8b1981cb50694a9113d94da7b782f41 not found: ID does not exist"
Sep 15 17:03:19.948391 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948374    2578 scope.go:117] "RemoveContainer" containerID="1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"
Sep 15 17:03:19.948647 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:19.948629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13\": container with ID starting with 1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13 not found: ID does not exist" containerID="1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"
Sep 15 17:03:19.948689 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948652    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13"} err="failed to get container status \"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13\": rpc error: code = NotFound desc = could not find container \"1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13\": container with ID starting with 1eef26dca965180d3bfef0120a274e0f5f411a36018997ea98e650ed3c396a13 not found: ID does not exist"
Sep 15 17:03:19.948689 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948668    2578 scope.go:117] "RemoveContainer" containerID="a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"
Sep 15 17:03:19.948919 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:19.948904    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297\": container with ID starting with a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297 not found: ID does not exist" containerID="a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"
Sep 15 17:03:19.948973 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:19.948924    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297"} err="failed to get container status \"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297\": rpc error: code = NotFound desc = could not find container \"a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297\": container with ID starting with a4cc450e8613a1457bf85f1452134e9a06254a6acc331fa90fb6ef9a821f0297 not found: ID does not exist"
Sep 15 17:03:20.004820 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.004794    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.004820 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.004817    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dj6h\" (UniqueName: \"kubernetes.io/projected/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-api-access-8dj6h\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.004987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.004830    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.004987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.004840    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cc852f3-7157-4499-ad1c-f88e6bff3e02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.004987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.004849    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cc852f3-7157-4499-ad1c-f88e6bff3e02-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.280325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280236    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:03:20.280686 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280667    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="storage-initializer"
Sep 15 17:03:20.280804 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280689    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="storage-initializer"
Sep 15 17:03:20.280804 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280718    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container"
Sep 15 17:03:20.280804 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280727    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container"
Sep 15 17:03:20.280804 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280765    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kube-rbac-proxy"
Sep 15 17:03:20.280804 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kube-rbac-proxy"
Sep 15 17:03:20.281091 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280871    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kube-rbac-proxy"
Sep 15 17:03:20.281091 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.280886    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" containerName="kserve-container"
Sep 15 17:03:20.285692 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.285668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.287762 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.287719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 15 17:03:20.287864 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.287725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 15 17:03:20.295359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.295339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:03:20.408438 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.408399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgxg\" (UniqueName: \"kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.408438 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.408444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.408640 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.408467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.408640 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.408579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.408640 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.408602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.509977 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.509942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgxg\" (UniqueName: \"kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510162 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.509985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510162 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510162 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510162 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510327 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:20.510279    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 17:03:20.510412 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:03:20.510398    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls podName:2ea72c61-c88c-404c-8ba2-33e84c49933b nodeName:}" failed. No retries permitted until 2026-09-15 17:03:21.010342574 +0000 UTC m=+2172.396981836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls") pod "isvc-canary-stop-predictor-7f6f9c568d-7djp6" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 17:03:20.510458 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510728 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.510794 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.510712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:20.520193 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:20.520168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgxg\" (UniqueName: \"kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:21.015555 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.015522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:21.018021 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.017998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-7djp6\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:21.154436 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.154403    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc852f3-7157-4499-ad1c-f88e6bff3e02" path="/var/lib/kubelet/pods/0cc852f3-7157-4499-ad1c-f88e6bff3e02/volumes"
Sep 15 17:03:21.197704 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.197673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:21.472120 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.472083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:03:21.475114 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:03:21.475078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea72c61_c88c_404c_8ba2_33e84c49933b.slice/crio-6e3a8c06859545097f3354f9f1b270a2730fbbaa7f51d6356bffb8d2dfce1802 WatchSource:0}: Error finding container 6e3a8c06859545097f3354f9f1b270a2730fbbaa7f51d6356bffb8d2dfce1802: Status 404 returned error can't find the container with id 6e3a8c06859545097f3354f9f1b270a2730fbbaa7f51d6356bffb8d2dfce1802
Sep 15 17:03:21.937863 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.937820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerStarted","Data":"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"}
Sep 15 17:03:21.938025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:21.937889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerStarted","Data":"6e3a8c06859545097f3354f9f1b270a2730fbbaa7f51d6356bffb8d2dfce1802"}
Sep 15 17:03:25.960464 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:25.960425    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerID="85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa" exitCode=0
Sep 15 17:03:25.960967 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:25.960474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerDied","Data":"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"}
Sep 15 17:03:26.968139 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:26.968104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerStarted","Data":"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"}
Sep 15 17:03:26.968544 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:26.968166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerStarted","Data":"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"}
Sep 15 17:03:26.968544 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:26.968353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:26.986602 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:26.986539    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podStartSLOduration=2.895147063 podStartE2EDuration="6.986519456s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.091372393s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:03:26.984941674 +0000 UTC m=+2178.371580952" watchObservedRunningTime="2026-09-15 17:03:26.986519456 +0000 UTC m=+2178.373158724"
Sep 15 17:03:27.973299 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:27.973267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:27.974537 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:27.974504    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:03:28.977878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:28.977845    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:03:33.982582 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:33.982550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:03:33.983115 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:33.983090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:03:43.983044 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:43.983004    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:03:47.897191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:47.897155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:03:53.983962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:53.983922    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:03:54.807223 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:54.807188    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:03:54.807477 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:54.807453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kserve-container" containerID="cri-o://9fa75689aa167ebc1b2eaf56e9718616c817505e151b67a34bbf00abbead0495" gracePeriod=30
Sep 15 17:03:54.807662 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:54.807611    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kube-rbac-proxy" containerID="cri-o://1e710b056715347a981796e22167f1bf19bc3205ac68bedbe2afcd81f41b44c9" gracePeriod=30
Sep 15 17:03:55.109796 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:55.109691    2578 generic.go:356] "Generic (PLEG): container finished" podID="c491107d-b693-4efd-a928-4a8a3532702e" containerID="1e710b056715347a981796e22167f1bf19bc3205ac68bedbe2afcd81f41b44c9" exitCode=2
Sep 15 17:03:55.109796 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:55.109772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerDied","Data":"1e710b056715347a981796e22167f1bf19bc3205ac68bedbe2afcd81f41b44c9"}
Sep 15 17:03:57.888133 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:57.888072    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 15 17:03:57.894155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:03:57.894118    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 15 17:04:00.012062 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.012023    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:00.015915 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.015898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.018115 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.018091    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 15 17:04:00.018238 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.018167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 15 17:04:00.025505 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.025473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:00.056390 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.056362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.056522 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.056407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.056522 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.056429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.056522 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.056512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.056644 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.056544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.157798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.157767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.157964 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.157808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.157964 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.157859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.158055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.157995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.158055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.158030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.158193 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:00.158176    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 15 17:04:00.158252 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.158227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.158317 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:00.158234    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls podName:ef761c4e-2b11-4500-a3c8-7a8c489889ab nodeName:}" failed. No retries permitted until 2026-09-15 17:04:00.658213092 +0000 UTC m=+2212.044852351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 15 17:04:00.158542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.158522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.158731 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.158710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.168272 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.168254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.662959 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.662916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.665464 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.665432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:00.927279 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:00.927182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:01.140488 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.140460    2578 generic.go:356] "Generic (PLEG): container finished" podID="c491107d-b693-4efd-a928-4a8a3532702e" containerID="9fa75689aa167ebc1b2eaf56e9718616c817505e151b67a34bbf00abbead0495" exitCode=0
Sep 15 17:04:01.140802 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.140532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerDied","Data":"9fa75689aa167ebc1b2eaf56e9718616c817505e151b67a34bbf00abbead0495"}
Sep 15 17:04:01.200978 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.200886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:01.203141 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:04:01.203110    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef761c4e_2b11_4500_a3c8_7a8c489889ab.slice/crio-b522e42e2c4723d13ccc288a3a9834aad74dcb638653bc417aab90b10494b159 WatchSource:0}: Error finding container b522e42e2c4723d13ccc288a3a9834aad74dcb638653bc417aab90b10494b159: Status 404 returned error can't find the container with id b522e42e2c4723d13ccc288a3a9834aad74dcb638653bc417aab90b10494b159
Sep 15 17:04:01.273328 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.273301    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:04:01.369017 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.368988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location\") pod \"c491107d-b693-4efd-a928-4a8a3532702e\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") "
Sep 15 17:04:01.369180 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config\") pod \"c491107d-b693-4efd-a928-4a8a3532702e\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") "
Sep 15 17:04:01.369180 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert\") pod \"c491107d-b693-4efd-a928-4a8a3532702e\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") "
Sep 15 17:04:01.369180 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxxkh\" (UniqueName: \"kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh\") pod \"c491107d-b693-4efd-a928-4a8a3532702e\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") "
Sep 15 17:04:01.369320 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls\") pod \"c491107d-b693-4efd-a928-4a8a3532702e\" (UID: \"c491107d-b693-4efd-a928-4a8a3532702e\") "
Sep 15 17:04:01.369378 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c491107d-b693-4efd-a928-4a8a3532702e" (UID: "c491107d-b693-4efd-a928-4a8a3532702e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:04:01.369473 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c491107d-b693-4efd-a928-4a8a3532702e" (UID: "c491107d-b693-4efd-a928-4a8a3532702e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:01.369536 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369504    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491107d-b693-4efd-a928-4a8a3532702e-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:01.369601 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.369566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c491107d-b693-4efd-a928-4a8a3532702e" (UID: "c491107d-b693-4efd-a928-4a8a3532702e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:01.371434 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.371403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c491107d-b693-4efd-a928-4a8a3532702e" (UID: "c491107d-b693-4efd-a928-4a8a3532702e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:04:01.371590 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.371567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh" (OuterVolumeSpecName: "kube-api-access-gxxkh") pod "c491107d-b693-4efd-a928-4a8a3532702e" (UID: "c491107d-b693-4efd-a928-4a8a3532702e"). InnerVolumeSpecName "kube-api-access-gxxkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:04:01.470648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.470565    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:01.470648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.470594    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491107d-b693-4efd-a928-4a8a3532702e-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:01.470648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.470603    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxxkh\" (UniqueName: \"kubernetes.io/projected/c491107d-b693-4efd-a928-4a8a3532702e-kube-api-access-gxxkh\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:01.470648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:01.470613    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491107d-b693-4efd-a928-4a8a3532702e-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:02.152648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.152601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt" event={"ID":"c491107d-b693-4efd-a928-4a8a3532702e","Type":"ContainerDied","Data":"9541f10b947848524ace9bbc7388ea8b4ccdad2f4cb03a611c745cf8752f3596"}
Sep 15 17:04:02.153138 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.152855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"
Sep 15 17:04:02.153138 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.152864    2578 scope.go:117] "RemoveContainer" containerID="1e710b056715347a981796e22167f1bf19bc3205ac68bedbe2afcd81f41b44c9"
Sep 15 17:04:02.154930 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.154906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerStarted","Data":"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"}
Sep 15 17:04:02.155045 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.154969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerStarted","Data":"b522e42e2c4723d13ccc288a3a9834aad74dcb638653bc417aab90b10494b159"}
Sep 15 17:04:02.162450 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.162429    2578 scope.go:117] "RemoveContainer" containerID="9fa75689aa167ebc1b2eaf56e9718616c817505e151b67a34bbf00abbead0495"
Sep 15 17:04:02.170257 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.170238    2578 scope.go:117] "RemoveContainer" containerID="78d0025b09b000f8afe2d75ab3e5189337d7ec6082abae901ceb5e7dbe6365a1"
Sep 15 17:04:02.188912 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.188883    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:04:02.192191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:02.192171    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wmjkt"]
Sep 15 17:04:03.154519 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:03.154485    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c491107d-b693-4efd-a928-4a8a3532702e" path="/var/lib/kubelet/pods/c491107d-b693-4efd-a928-4a8a3532702e/volumes"
Sep 15 17:04:03.983965 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:03.983926    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:04:05.171145 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:05.171108    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerID="e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589" exitCode=0
Sep 15 17:04:05.171626 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:05.171153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerDied","Data":"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"}
Sep 15 17:04:06.177893 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:06.177856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerStarted","Data":"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"}
Sep 15 17:04:06.178380 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:06.177928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerStarted","Data":"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"}
Sep 15 17:04:06.178380 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:06.178143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:06.198661 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:06.198609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" podStartSLOduration=3.525839869 podStartE2EDuration="7.19859361s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672753741s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:04:06.195934475 +0000 UTC m=+2217.582573757" watchObservedRunningTime="2026-09-15 17:04:06.19859361 +0000 UTC m=+2217.585232877"
Sep 15 17:04:07.182338 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:07.182307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:13.193937 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:13.193904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:13.983028 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:13.982990    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:04:23.983356 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:23.983319    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:04:33.984075 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:33.983995    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 15 17:04:42.370613 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:42.370579    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:04:42.371039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:42.370834    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container" containerID="cri-o://774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4" gracePeriod=30
Sep 15 17:04:42.371039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:42.370928    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kube-rbac-proxy" containerID="cri-o://58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955" gracePeriod=30
Sep 15 17:04:42.371883 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:42.371724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:04:43.197984 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:43.197955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:43.355846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:43.355810    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerID="58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955" exitCode=2
Sep 15 17:04:43.355846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:43.355846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerDied","Data":"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"}
Sep 15 17:04:43.978829 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:43.978791    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 15 17:04:47.014146 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.014121    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:04:47.067611 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert\") pod \"2ea72c61-c88c-404c-8ba2-33e84c49933b\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") "
Sep 15 17:04:47.067611 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") pod \"2ea72c61-c88c-404c-8ba2-33e84c49933b\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") "
Sep 15 17:04:47.067877 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcgxg\" (UniqueName: \"kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg\") pod \"2ea72c61-c88c-404c-8ba2-33e84c49933b\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") "
Sep 15 17:04:47.067877 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config\") pod \"2ea72c61-c88c-404c-8ba2-33e84c49933b\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") "
Sep 15 17:04:47.067877 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location\") pod \"2ea72c61-c88c-404c-8ba2-33e84c49933b\" (UID: \"2ea72c61-c88c-404c-8ba2-33e84c49933b\") "
Sep 15 17:04:47.068047 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.067953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ea72c61-c88c-404c-8ba2-33e84c49933b" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:47.068123 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.068102    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:47.068191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.068124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ea72c61-c88c-404c-8ba2-33e84c49933b" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:04:47.068191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.068142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ea72c61-c88c-404c-8ba2-33e84c49933b" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:47.069957 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.069937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg" (OuterVolumeSpecName: "kube-api-access-vcgxg") pod "2ea72c61-c88c-404c-8ba2-33e84c49933b" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b"). InnerVolumeSpecName "kube-api-access-vcgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:04:47.070064 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.070046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ea72c61-c88c-404c-8ba2-33e84c49933b" (UID: "2ea72c61-c88c-404c-8ba2-33e84c49933b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:04:47.169472 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.169447    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:47.169472 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.169472    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea72c61-c88c-404c-8ba2-33e84c49933b-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:47.169625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.169481    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea72c61-c88c-404c-8ba2-33e84c49933b-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:47.169625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.169490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcgxg\" (UniqueName: \"kubernetes.io/projected/2ea72c61-c88c-404c-8ba2-33e84c49933b-kube-api-access-vcgxg\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:47.382474 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.382365    2578 generic.go:356] "Generic (PLEG): container finished" podID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerID="774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4" exitCode=0
Sep 15 17:04:47.382636 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.382511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerDied","Data":"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"}
Sep 15 17:04:47.382636 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.382553    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"
Sep 15 17:04:47.382790 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.382653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6" event={"ID":"2ea72c61-c88c-404c-8ba2-33e84c49933b","Type":"ContainerDied","Data":"6e3a8c06859545097f3354f9f1b270a2730fbbaa7f51d6356bffb8d2dfce1802"}
Sep 15 17:04:47.382790 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.382706    2578 scope.go:117] "RemoveContainer" containerID="58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"
Sep 15 17:04:47.392178 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.392150    2578 scope.go:117] "RemoveContainer" containerID="774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"
Sep 15 17:04:47.400279 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.400177    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:04:47.401237 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.401218    2578 scope.go:117] "RemoveContainer" containerID="85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"
Sep 15 17:04:47.403347 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.403326    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-7djp6"]
Sep 15 17:04:47.409058 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409037    2578 scope.go:117] "RemoveContainer" containerID="58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"
Sep 15 17:04:47.409352 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:47.409333    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955\": container with ID starting with 58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955 not found: ID does not exist" containerID="58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"
Sep 15 17:04:47.409412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409362    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955"} err="failed to get container status \"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955\": rpc error: code = NotFound desc = could not find container \"58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955\": container with ID starting with 58fcd124894d4ecf090d3a87180267cf60bede85efb51904ae60fe44be24f955 not found: ID does not exist"
Sep 15 17:04:47.409412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409389    2578 scope.go:117] "RemoveContainer" containerID="774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"
Sep 15 17:04:47.409643 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:47.409627    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4\": container with ID starting with 774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4 not found: ID does not exist" containerID="774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"
Sep 15 17:04:47.409688 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409649    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4"} err="failed to get container status \"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4\": rpc error: code = NotFound desc = could not find container \"774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4\": container with ID starting with 774de21af01abebd20b06b323f85f4f43e1eaf7c18a9ce4e7bddea7856eb9ba4 not found: ID does not exist"
Sep 15 17:04:47.409688 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409663    2578 scope.go:117] "RemoveContainer" containerID="85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"
Sep 15 17:04:47.409945 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:47.409926    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa\": container with ID starting with 85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa not found: ID does not exist" containerID="85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"
Sep 15 17:04:47.409987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:47.409952    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa"} err="failed to get container status \"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa\": rpc error: code = NotFound desc = could not find container \"85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa\": container with ID starting with 85ec95fb94c6247dd9acce70d1c739ac63c43e2cb80421315708ecc4f0e2e4fa not found: ID does not exist"
Sep 15 17:04:49.153840 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:49.153811    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" path="/var/lib/kubelet/pods/2ea72c61-c88c-404c-8ba2-33e84c49933b/volumes"
Sep 15 17:04:50.053122 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:50.053083    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:50.053386 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:50.053354    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kserve-container" containerID="cri-o://c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064" gracePeriod=30
Sep 15 17:04:50.053474 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:50.053440    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kube-rbac-proxy" containerID="cri-o://3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971" gracePeriod=30
Sep 15 17:04:50.399198 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:50.399170    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerID="3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971" exitCode=2
Sep 15 17:04:50.399571 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:50.399243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerDied","Data":"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"}
Sep 15 17:04:53.188238 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:53.188197    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 15 17:04:53.194526 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:53.194488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 15 17:04:56.295370 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.295348    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:56.349998 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.349922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config\") pod \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") "
Sep 15 17:04:56.349998 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.349992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6\") pod \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") "
Sep 15 17:04:56.350217 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert\") pod \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") "
Sep 15 17:04:56.350217 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") pod \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") "
Sep 15 17:04:56.350217 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location\") pod \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\" (UID: \"ef761c4e-2b11-4500-a3c8-7a8c489889ab\") "
Sep 15 17:04:56.350375 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef761c4e-2b11-4500-a3c8-7a8c489889ab" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:56.350443 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef761c4e-2b11-4500-a3c8-7a8c489889ab" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:56.350504 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350492    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:56.350559 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350510    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:56.350615 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.350593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef761c4e-2b11-4500-a3c8-7a8c489889ab" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:04:56.352124 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.352103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6" (OuterVolumeSpecName: "kube-api-access-p56n6") pod "ef761c4e-2b11-4500-a3c8-7a8c489889ab" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab"). InnerVolumeSpecName "kube-api-access-p56n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:04:56.352260 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.352237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef761c4e-2b11-4500-a3c8-7a8c489889ab" (UID: "ef761c4e-2b11-4500-a3c8-7a8c489889ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:04:56.427710 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.427682    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerID="c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064" exitCode=0
Sep 15 17:04:56.427914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.427760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerDied","Data":"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"}
Sep 15 17:04:56.427914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.427774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"
Sep 15 17:04:56.427914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.427818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq" event={"ID":"ef761c4e-2b11-4500-a3c8-7a8c489889ab","Type":"ContainerDied","Data":"b522e42e2c4723d13ccc288a3a9834aad74dcb638653bc417aab90b10494b159"}
Sep 15 17:04:56.427914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.427847    2578 scope.go:117] "RemoveContainer" containerID="3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"
Sep 15 17:04:56.436983 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.436967    2578 scope.go:117] "RemoveContainer" containerID="c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"
Sep 15 17:04:56.444837 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.444819    2578 scope.go:117] "RemoveContainer" containerID="e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"
Sep 15 17:04:56.449093 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.449073    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:56.451298 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.451277    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kube-api-access-p56n6\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:56.451384 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.451298    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef761c4e-2b11-4500-a3c8-7a8c489889ab-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:56.451384 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.451315    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef761c4e-2b11-4500-a3c8-7a8c489889ab-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:04:56.452870 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.452857    2578 scope.go:117] "RemoveContainer" containerID="3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"
Sep 15 17:04:56.453136 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:56.453117    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971\": container with ID starting with 3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971 not found: ID does not exist" containerID="3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"
Sep 15 17:04:56.453200 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.453146    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971"} err="failed to get container status \"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971\": rpc error: code = NotFound desc = could not find container \"3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971\": container with ID starting with 3e1d75065ffb6a1e93fd7067fedc4527514a9aa711a304c9e4a9ead84bf5d971 not found: ID does not exist"
Sep 15 17:04:56.453200 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.453163    2578 scope.go:117] "RemoveContainer" containerID="c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"
Sep 15 17:04:56.453408 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:56.453394    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064\": container with ID starting with c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064 not found: ID does not exist" containerID="c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"
Sep 15 17:04:56.453454 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.453412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064"} err="failed to get container status \"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064\": rpc error: code = NotFound desc = could not find container \"c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064\": container with ID starting with c9408728f849aabea8a1c0075a2625f140dbbae51b32157233ea49b53daa7064 not found: ID does not exist"
Sep 15 17:04:56.453454 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.453424    2578 scope.go:117] "RemoveContainer" containerID="e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"
Sep 15 17:04:56.453654 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:04:56.453639    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589\": container with ID starting with e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589 not found: ID does not exist" containerID="e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"
Sep 15 17:04:56.453697 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.453657    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589"} err="failed to get container status \"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589\": rpc error: code = NotFound desc = could not find container \"e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589\": container with ID starting with e2ee75db6b68df10ccc0ecf2f562b2f2356109276cfbd0f254142363f265d589 not found: ID does not exist"
Sep 15 17:04:56.456763 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:56.456721    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bfqmq"]
Sep 15 17:04:57.154587 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:04:57.154558    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" path="/var/lib/kubelet/pods/ef761c4e-2b11-4500-a3c8-7a8c489889ab/volumes"
Sep 15 17:06:40.546085 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546425    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546452    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="storage-initializer"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="storage-initializer"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546470    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546476    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546482    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="storage-initializer"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546487    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="storage-initializer"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546497    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546502    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kube-rbac-proxy"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546508    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546513    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546519    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kserve-container"
Sep 15 17:06:40.546534 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="storage-initializer"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546548    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="storage-initializer"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546598    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kserve-container"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546609    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kube-rbac-proxy"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546621    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef761c4e-2b11-4500-a3c8-7a8c489889ab" containerName="kube-rbac-proxy"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546628    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kube-rbac-proxy"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546634    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c491107d-b693-4efd-a928-4a8a3532702e" containerName="kserve-container"
Sep 15 17:06:40.547079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.546639    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea72c61-c88c-404c-8ba2-33e84c49933b" containerName="kserve-container"
Sep 15 17:06:40.549960 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.549944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.552307 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.552283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 15 17:06:40.552307 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.552299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:06:40.552489 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.552346    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:06:40.552561 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.552547    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:06:40.553201 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.553187    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:06:40.553253 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.553213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:06:40.553301 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.553255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:06:40.559707 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.559688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:06:40.631508 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.631472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.631681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.631516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.631681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.631540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6l6s\" (UniqueName: \"kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.631681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.631591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.631681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.631658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.732917 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.732885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733070 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.732949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733070 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.732976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733070 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:06:40.733054    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 15 17:06:40.733218 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.733073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6l6s\" (UniqueName: \"kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733218 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:06:40.733117    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls podName:d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5 nodeName:}" failed. No retries permitted until 2026-09-15 17:06:41.233095179 +0000 UTC m=+2372.619734444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 15 17:06:40.733218 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.733137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.733424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733639 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.733621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.733682 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.733643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:40.743524 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:40.743501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6l6s\" (UniqueName: \"kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:41.238265 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.238225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:41.240763 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.240722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:41.460578 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.460544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:41.734685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.734658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:06:41.737059 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:06:41.737033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd164c8ec_1f8e_47a9_96ef_6573e7b8b6c5.slice/crio-24d675e58d324fea4383a87c21646e587ce094f64bbedfea1f11ec78d0ebfc3d WatchSource:0}: Error finding container 24d675e58d324fea4383a87c21646e587ce094f64bbedfea1f11ec78d0ebfc3d: Status 404 returned error can't find the container with id 24d675e58d324fea4383a87c21646e587ce094f64bbedfea1f11ec78d0ebfc3d
Sep 15 17:06:41.873404 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.873347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerStarted","Data":"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"}
Sep 15 17:06:41.873603 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:41.873428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerStarted","Data":"24d675e58d324fea4383a87c21646e587ce094f64bbedfea1f11ec78d0ebfc3d"}
Sep 15 17:06:45.892866 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:45.892827    2578 generic.go:356] "Generic (PLEG): container finished" podID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerID="ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a" exitCode=0
Sep 15 17:06:45.892866 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:45.892863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerDied","Data":"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"}
Sep 15 17:06:46.900929 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:46.900900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerStarted","Data":"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"}
Sep 15 17:06:46.901316 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:46.900957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerStarted","Data":"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"}
Sep 15 17:06:46.901316 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:46.901169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:46.901460 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:46.901328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:06:46.921839 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:46.921789    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" podStartSLOduration=3.176006787 podStartE2EDuration="6.921772001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745765214s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:06:46.920842945 +0000 UTC m=+2378.307482213" watchObservedRunningTime="2026-09-15 17:06:46.921772001 +0000 UTC m=+2378.308411269"
Sep 15 17:06:52.912513 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:06:52.912484    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:07:22.988138 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:22.988092    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 15 17:07:32.915392 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:32.915309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:07:40.644871 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:40.644837    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:07:40.645342 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:40.645142    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kserve-container" containerID="cri-o://59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2" gracePeriod=30
Sep 15 17:07:40.645342 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:40.645168    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kube-rbac-proxy" containerID="cri-o://876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f" gracePeriod=30
Sep 15 17:07:41.139689 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:41.139657    2578 generic.go:356] "Generic (PLEG): container finished" podID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerID="876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f" exitCode=2
Sep 15 17:07:41.139854 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:41.139731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerDied","Data":"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"}
Sep 15 17:07:42.906869 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:42.906825    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 15 17:07:47.688299 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.688276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:07:47.834013 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.833930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6l6s\" (UniqueName: \"kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s\") pod \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") "
Sep 15 17:07:47.834013 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.833981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location\") pod \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") "
Sep 15 17:07:47.834247 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert\") pod \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") "
Sep 15 17:07:47.834247 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config\") pod \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") "
Sep 15 17:07:47.834247 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") pod \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\" (UID: \"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5\") "
Sep 15 17:07:47.834419 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:07:47.834481 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:07:47.834481 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.834441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:07:47.836352 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.836327    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s" (OuterVolumeSpecName: "kube-api-access-v6l6s") pod "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5"). InnerVolumeSpecName "kube-api-access-v6l6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:07:47.836426 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.836353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" (UID: "d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:07:47.934610 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.934575    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6l6s\" (UniqueName: \"kubernetes.io/projected/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-api-access-v6l6s\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:07:47.934610 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.934610    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:07:47.934857 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.934621    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:07:47.934857 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.934631    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:07:47.934857 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:47.934660    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:07:48.170048 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.170015    2578 generic.go:356] "Generic (PLEG): container finished" podID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerID="59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2" exitCode=0
Sep 15 17:07:48.170243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.170066    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"
Sep 15 17:07:48.170243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.170080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerDied","Data":"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"}
Sep 15 17:07:48.170243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.170134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt" event={"ID":"d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5","Type":"ContainerDied","Data":"24d675e58d324fea4383a87c21646e587ce094f64bbedfea1f11ec78d0ebfc3d"}
Sep 15 17:07:48.170243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.170169    2578 scope.go:117] "RemoveContainer" containerID="876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"
Sep 15 17:07:48.180151 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.180131    2578 scope.go:117] "RemoveContainer" containerID="59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"
Sep 15 17:07:48.187879 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.187860    2578 scope.go:117] "RemoveContainer" containerID="ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"
Sep 15 17:07:48.191909 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.191886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:07:48.195108 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.195085    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-cpmpt"]
Sep 15 17:07:48.196351 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.196331    2578 scope.go:117] "RemoveContainer" containerID="876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"
Sep 15 17:07:48.196655 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:07:48.196637    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f\": container with ID starting with 876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f not found: ID does not exist" containerID="876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"
Sep 15 17:07:48.196700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.196664    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f"} err="failed to get container status \"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f\": rpc error: code = NotFound desc = could not find container \"876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f\": container with ID starting with 876ec7e5f5d35d3398c1b07bf038e01e1af3875ed4f92a905f7521c7a5aa274f not found: ID does not exist"
Sep 15 17:07:48.196700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.196680    2578 scope.go:117] "RemoveContainer" containerID="59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"
Sep 15 17:07:48.196970 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:07:48.196951    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2\": container with ID starting with 59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2 not found: ID does not exist" containerID="59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"
Sep 15 17:07:48.197038 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.196989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2"} err="failed to get container status \"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2\": rpc error: code = NotFound desc = could not find container \"59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2\": container with ID starting with 59cd391f0ea9b61f73dd3da9ece9dcd2eeec3da85acfc2c4a56ad43b807887e2 not found: ID does not exist"
Sep 15 17:07:48.197038 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.197010    2578 scope.go:117] "RemoveContainer" containerID="ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"
Sep 15 17:07:48.197289 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:07:48.197270    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a\": container with ID starting with ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a not found: ID does not exist" containerID="ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"
Sep 15 17:07:48.197338 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:48.197292    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a"} err="failed to get container status \"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a\": rpc error: code = NotFound desc = could not find container \"ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a\": container with ID starting with ddf5f3a1d805e405ac8ba96ea36a43993b577130e25f5fbdddbaa7ce5ba9fc5a not found: ID does not exist"
Sep 15 17:07:49.154982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:07:49.154953    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" path="/var/lib/kubelet/pods/d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5/volumes"
Sep 15 17:08:31.752959 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.752921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:08:31.753361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753304    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kserve-container"
Sep 15 17:08:31.753361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753317    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kserve-container"
Sep 15 17:08:31.753361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753330    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kube-rbac-proxy"
Sep 15 17:08:31.753361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753335    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kube-rbac-proxy"
Sep 15 17:08:31.753361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="storage-initializer"
Sep 15 17:08:31.753531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="storage-initializer"
Sep 15 17:08:31.753531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753424    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kserve-container"
Sep 15 17:08:31.753531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.753438    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d164c8ec-1f8e-47a9-96ef-6573e7b8b6c5" containerName="kube-rbac-proxy"
Sep 15 17:08:31.756806 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.756787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:31.759303 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.759268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:08:31.759303 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.759287    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:08:31.759523 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.759287    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:08:31.759523 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.759422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:08:31.759619 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.759605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:08:31.760143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.760123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 15 17:08:31.760196 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.760163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:08:31.768714 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.768689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:08:31.907052 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.907018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:31.907258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.907112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:31.907258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.907167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:31.907258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.907224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:31.907258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:31.907247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76hh\" (UniqueName: \"kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008074 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.007986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76hh\" (UniqueName: \"kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008074 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008309 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008309 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008309 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008628 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008842 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.008911 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.008857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.010752 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.010710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.016095 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.016070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76hh\" (UniqueName: \"kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.068179 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.068140    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:32.340006 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.339982    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:08:32.342211 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:08:32.342178    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e86775_8910_4888_b321_fd76f6c261a4.slice/crio-d97c35b2fe9821ba9537070364859756e07213345b5552374cbb5628dcb0ba11 WatchSource:0}: Error finding container d97c35b2fe9821ba9537070364859756e07213345b5552374cbb5628dcb0ba11: Status 404 returned error can't find the container with id d97c35b2fe9821ba9537070364859756e07213345b5552374cbb5628dcb0ba11
Sep 15 17:08:32.365849 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:32.365824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerStarted","Data":"d97c35b2fe9821ba9537070364859756e07213345b5552374cbb5628dcb0ba11"}
Sep 15 17:08:33.372388 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:33.372348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerStarted","Data":"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"}
Sep 15 17:08:36.257804 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:08:36.257766    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e86775_8910_4888_b321_fd76f6c261a4.slice/crio-conmon-cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 17:08:36.387953 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:36.387917    2578 generic.go:356] "Generic (PLEG): container finished" podID="68e86775-8910-4888-b321-fd76f6c261a4" containerID="cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462" exitCode=0
Sep 15 17:08:36.388111 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:36.387959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerDied","Data":"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"}
Sep 15 17:08:37.394313 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:37.394276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerStarted","Data":"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"}
Sep 15 17:08:37.394719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:37.394349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerStarted","Data":"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"}
Sep 15 17:08:37.394719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:37.394552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:37.394719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:37.394606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:08:37.413301 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:37.413238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" podStartSLOduration=2.60167348 podStartE2EDuration="6.41321625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81154277s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:08:37.411876684 +0000 UTC m=+2488.798515953" watchObservedRunningTime="2026-09-15 17:08:37.41321625 +0000 UTC m=+2488.799855527"
Sep 15 17:08:43.405239 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:08:43.405208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:09:13.409351 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:13.409311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:09:21.903246 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:21.903215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:09:21.903611 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:21.903499    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kserve-container" containerID="cri-o://78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7" gracePeriod=30
Sep 15 17:09:21.903676 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:21.903607    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kube-rbac-proxy" containerID="cri-o://5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc" gracePeriod=30
Sep 15 17:09:22.594763 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:22.594704    2578 generic.go:356] "Generic (PLEG): container finished" podID="68e86775-8910-4888-b321-fd76f6c261a4" containerID="5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc" exitCode=2
Sep 15 17:09:22.594939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:22.594775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerDied","Data":"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"}
Sep 15 17:09:22.945875 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:22.945848    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:09:23.041134 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location\") pod \"68e86775-8910-4888-b321-fd76f6c261a4\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") "
Sep 15 17:09:23.041273 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041188    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls\") pod \"68e86775-8910-4888-b321-fd76f6c261a4\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") "
Sep 15 17:09:23.041273 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config\") pod \"68e86775-8910-4888-b321-fd76f6c261a4\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") "
Sep 15 17:09:23.041389 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h76hh\" (UniqueName: \"kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh\") pod \"68e86775-8910-4888-b321-fd76f6c261a4\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") "
Sep 15 17:09:23.041389 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert\") pod \"68e86775-8910-4888-b321-fd76f6c261a4\" (UID: \"68e86775-8910-4888-b321-fd76f6c261a4\") "
Sep 15 17:09:23.041589 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68e86775-8910-4888-b321-fd76f6c261a4" (UID: "68e86775-8910-4888-b321-fd76f6c261a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:09:23.041713 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68e86775-8910-4888-b321-fd76f6c261a4" (UID: "68e86775-8910-4888-b321-fd76f6c261a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:23.041795 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.041758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68e86775-8910-4888-b321-fd76f6c261a4" (UID: "68e86775-8910-4888-b321-fd76f6c261a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:23.043476 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.043457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh" (OuterVolumeSpecName: "kube-api-access-h76hh") pod "68e86775-8910-4888-b321-fd76f6c261a4" (UID: "68e86775-8910-4888-b321-fd76f6c261a4"). InnerVolumeSpecName "kube-api-access-h76hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:09:23.043542 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.043507    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68e86775-8910-4888-b321-fd76f6c261a4" (UID: "68e86775-8910-4888-b321-fd76f6c261a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:09:23.142490 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.142459    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68e86775-8910-4888-b321-fd76f6c261a4-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:09:23.142490 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.142486    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:09:23.142490 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.142495    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h76hh\" (UniqueName: \"kubernetes.io/projected/68e86775-8910-4888-b321-fd76f6c261a4-kube-api-access-h76hh\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:09:23.142712 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.142504    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68e86775-8910-4888-b321-fd76f6c261a4-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:09:23.142712 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.142513    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e86775-8910-4888-b321-fd76f6c261a4-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:09:23.601081 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.600992    2578 generic.go:356] "Generic (PLEG): container finished" podID="68e86775-8910-4888-b321-fd76f6c261a4" containerID="78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7" exitCode=0
Sep 15 17:09:23.601081 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.601047    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"
Sep 15 17:09:23.601334 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.601085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerDied","Data":"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"}
Sep 15 17:09:23.601334 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.601160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr" event={"ID":"68e86775-8910-4888-b321-fd76f6c261a4","Type":"ContainerDied","Data":"d97c35b2fe9821ba9537070364859756e07213345b5552374cbb5628dcb0ba11"}
Sep 15 17:09:23.601334 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.601197    2578 scope.go:117] "RemoveContainer" containerID="5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"
Sep 15 17:09:23.616174 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.616153    2578 scope.go:117] "RemoveContainer" containerID="78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"
Sep 15 17:09:23.621604 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.621577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:09:23.624449 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.624385    2578 scope.go:117] "RemoveContainer" containerID="cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"
Sep 15 17:09:23.625181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.625162    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-qdpvr"]
Sep 15 17:09:23.631955 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.631936    2578 scope.go:117] "RemoveContainer" containerID="5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"
Sep 15 17:09:23.632236 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:09:23.632218    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc\": container with ID starting with 5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc not found: ID does not exist" containerID="5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"
Sep 15 17:09:23.632319 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.632245    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc"} err="failed to get container status \"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc\": rpc error: code = NotFound desc = could not find container \"5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc\": container with ID starting with 5d7901a615ff53aa3fe23a6f6ad97d85235b38e06f5be3233687e7bd8cfd7fcc not found: ID does not exist"
Sep 15 17:09:23.632319 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.632268    2578 scope.go:117] "RemoveContainer" containerID="78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"
Sep 15 17:09:23.632493 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:09:23.632474    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7\": container with ID starting with 78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7 not found: ID does not exist" containerID="78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"
Sep 15 17:09:23.632546 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.632499    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7"} err="failed to get container status \"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7\": rpc error: code = NotFound desc = could not find container \"78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7\": container with ID starting with 78d8e60d87cacd16e0d0282f2d1bcb77dcaa80ebc38dd676876c001c435c3dd7 not found: ID does not exist"
Sep 15 17:09:23.632546 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.632515    2578 scope.go:117] "RemoveContainer" containerID="cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"
Sep 15 17:09:23.632840 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:09:23.632728    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462\": container with ID starting with cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462 not found: ID does not exist" containerID="cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"
Sep 15 17:09:23.632887 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:23.632848    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462"} err="failed to get container status \"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462\": rpc error: code = NotFound desc = could not find container \"cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462\": container with ID starting with cac4c0245492848636e1f7211c3fb0c3d489e4ec5f27b8590933a4ba25a2b462 not found: ID does not exist"
Sep 15 17:09:25.155320 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:25.155288    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e86775-8910-4888-b321-fd76f6c261a4" path="/var/lib/kubelet/pods/68e86775-8910-4888-b321-fd76f6c261a4/volumes"
Sep 15 17:09:52.333382 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333747    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kserve-container"
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333764    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kserve-container"
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333779    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="storage-initializer"
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333786    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="storage-initializer"
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333801    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kube-rbac-proxy"
Sep 15 17:09:52.333885 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333808    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kube-rbac-proxy"
Sep 15 17:09:52.334084 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333889    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kube-rbac-proxy"
Sep 15 17:09:52.334084 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.333901    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e86775-8910-4888-b321-fd76f6c261a4" containerName="kserve-container"
Sep 15 17:09:52.337213 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.337195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.339882 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.339858    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:09:52.340025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.339887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:09:52.340025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.339914    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:09:52.340025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.339920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:09:52.340025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.339858    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:09:52.340485 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.340469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:09:52.340530 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.340487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 15 17:09:52.345268 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.345246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:09:52.370394 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.370362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85vs\" (UniqueName: \"kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.370539 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.370405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.370539 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.370433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.370539 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.370518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.370688 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.370612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471429 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85vs\" (UniqueName: \"kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.471843 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:09:52.471632    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 15 17:09:52.471843 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:09:52.471764    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls podName:9efa8645-0553-4f6f-9d19-cf758e896a10 nodeName:}" failed. No retries permitted until 2026-09-15 17:09:52.971698044 +0000 UTC m=+2564.358337302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 15 17:09:52.471990 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.471970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.472195 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.472179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.472235 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.472178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.479907 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.479883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85vs\" (UniqueName: \"kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.976176 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.976126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:52.978646 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:52.978625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:53.248765 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:53.248665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:53.521018 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:53.520995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:09:53.522848 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:09:53.522823    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efa8645_0553_4f6f_9d19_cf758e896a10.slice/crio-9808296354dc1fddca1b042f5a8671baa89d835f4dcf14df8106d6d6b9809a10 WatchSource:0}: Error finding container 9808296354dc1fddca1b042f5a8671baa89d835f4dcf14df8106d6d6b9809a10: Status 404 returned error can't find the container with id 9808296354dc1fddca1b042f5a8671baa89d835f4dcf14df8106d6d6b9809a10
Sep 15 17:09:53.736719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:53.736679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerStarted","Data":"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"}
Sep 15 17:09:53.736921 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:53.736769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerStarted","Data":"9808296354dc1fddca1b042f5a8671baa89d835f4dcf14df8106d6d6b9809a10"}
Sep 15 17:09:58.759755 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:58.759706    2578 generic.go:356] "Generic (PLEG): container finished" podID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerID="8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626" exitCode=0
Sep 15 17:09:58.760151 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:58.759785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerDied","Data":"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"}
Sep 15 17:09:59.766166 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:59.766134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerStarted","Data":"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"}
Sep 15 17:09:59.766591 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:59.766193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerStarted","Data":"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"}
Sep 15 17:09:59.766591 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:59.766385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:09:59.786175 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:09:59.786125    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" podStartSLOduration=3.446287539 podStartE2EDuration="7.786104919s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.33981738s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:09:59.784590716 +0000 UTC m=+2571.171229984" watchObservedRunningTime="2026-09-15 17:09:59.786104919 +0000 UTC m=+2571.172744187"
Sep 15 17:10:00.770822 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:00.770792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:10:06.779576 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:06.779548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:10:36.783922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:36.783846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:10:41.297888 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.297853    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:10:41.301999 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.301981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.304269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.304243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 15 17:10:41.304269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.304263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:10:41.304453 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.304263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 15 17:10:41.311961 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.311937    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:10:41.399113 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.399078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.399113 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.399116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.399328 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.399141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.399328 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.399193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz962\" (UniqueName: \"kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.399328 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.399228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.500634 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.500601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.500861 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.500644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.500861 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.500668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.500861 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.500798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz962\" (UniqueName: \"kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.500861 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.500855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.501093 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.501073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.501452 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.501434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.501452 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.501445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.503348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.503332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.508483 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.508465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz962\" (UniqueName: \"kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.614269 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.614191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:41.889105 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.889079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:10:41.891868 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:10:41.891843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b1006_cd1d_414f_8f8b_329c98959eba.slice/crio-7c6bcf97402cb1be5292ac02b3e4160088f503f95c56b612b69b07fabb5222e5 WatchSource:0}: Error finding container 7c6bcf97402cb1be5292ac02b3e4160088f503f95c56b612b69b07fabb5222e5: Status 404 returned error can't find the container with id 7c6bcf97402cb1be5292ac02b3e4160088f503f95c56b612b69b07fabb5222e5
Sep 15 17:10:41.952878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:41.952848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerStarted","Data":"7c6bcf97402cb1be5292ac02b3e4160088f503f95c56b612b69b07fabb5222e5"}
Sep 15 17:10:42.379549 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.379466    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:10:42.379962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.379727    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kserve-container" containerID="cri-o://d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6" gracePeriod=30
Sep 15 17:10:42.379962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.379810    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kube-rbac-proxy" containerID="cri-o://75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c" gracePeriod=30
Sep 15 17:10:42.959602 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.959571    2578 generic.go:356] "Generic (PLEG): container finished" podID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerID="75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c" exitCode=2
Sep 15 17:10:42.959831 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.959649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerDied","Data":"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"}
Sep 15 17:10:42.961425 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:42.961388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerStarted","Data":"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"}
Sep 15 17:10:43.626271 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.626246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:10:43.720412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert\") pod \"9efa8645-0553-4f6f-9d19-cf758e896a10\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") "
Sep 15 17:10:43.720565 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location\") pod \"9efa8645-0553-4f6f-9d19-cf758e896a10\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") "
Sep 15 17:10:43.720565 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config\") pod \"9efa8645-0553-4f6f-9d19-cf758e896a10\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") "
Sep 15 17:10:43.720565 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") pod \"9efa8645-0553-4f6f-9d19-cf758e896a10\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") "
Sep 15 17:10:43.720703 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v85vs\" (UniqueName: \"kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs\") pod \"9efa8645-0553-4f6f-9d19-cf758e896a10\" (UID: \"9efa8645-0553-4f6f-9d19-cf758e896a10\") "
Sep 15 17:10:43.720811 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9efa8645-0553-4f6f-9d19-cf758e896a10" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:10:43.720866 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9efa8645-0553-4f6f-9d19-cf758e896a10" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:10:43.720909 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.720853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9efa8645-0553-4f6f-9d19-cf758e896a10" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:10:43.721042 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.721027    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:10:43.721099 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.721047    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9efa8645-0553-4f6f-9d19-cf758e896a10-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:10:43.721099 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.721056    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:10:43.722713 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.722691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9efa8645-0553-4f6f-9d19-cf758e896a10" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:10:43.722858 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.722834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs" (OuterVolumeSpecName: "kube-api-access-v85vs") pod "9efa8645-0553-4f6f-9d19-cf758e896a10" (UID: "9efa8645-0553-4f6f-9d19-cf758e896a10"). InnerVolumeSpecName "kube-api-access-v85vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:10:43.822373 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.822328    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9efa8645-0553-4f6f-9d19-cf758e896a10-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:10:43.822373 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.822368    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85vs\" (UniqueName: \"kubernetes.io/projected/9efa8645-0553-4f6f-9d19-cf758e896a10-kube-api-access-v85vs\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:10:43.968037 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.968003    2578 generic.go:356] "Generic (PLEG): container finished" podID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerID="d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6" exitCode=0
Sep 15 17:10:43.968230 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.968070    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"
Sep 15 17:10:43.968230 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.968075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerDied","Data":"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"}
Sep 15 17:10:43.968230 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.968221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb" event={"ID":"9efa8645-0553-4f6f-9d19-cf758e896a10","Type":"ContainerDied","Data":"9808296354dc1fddca1b042f5a8671baa89d835f4dcf14df8106d6d6b9809a10"}
Sep 15 17:10:43.968395 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.968268    2578 scope.go:117] "RemoveContainer" containerID="75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"
Sep 15 17:10:43.969856 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.969831    2578 generic.go:356] "Generic (PLEG): container finished" podID="381b1006-cd1d-414f-8f8b-329c98959eba" containerID="db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1" exitCode=0
Sep 15 17:10:43.969984 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.969903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerDied","Data":"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"}
Sep 15 17:10:43.978798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.978778    2578 scope.go:117] "RemoveContainer" containerID="d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"
Sep 15 17:10:43.986962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.986938    2578 scope.go:117] "RemoveContainer" containerID="8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"
Sep 15 17:10:43.999113 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999094    2578 scope.go:117] "RemoveContainer" containerID="75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"
Sep 15 17:10:43.999385 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:10:43.999368    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c\": container with ID starting with 75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c not found: ID does not exist" containerID="75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"
Sep 15 17:10:43.999442 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999391    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c"} err="failed to get container status \"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c\": rpc error: code = NotFound desc = could not find container \"75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c\": container with ID starting with 75cf7ee5db2c35bf034668fcc487840bf475f203c4e45ff192977797d7c6399c not found: ID does not exist"
Sep 15 17:10:43.999442 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999410    2578 scope.go:117] "RemoveContainer" containerID="d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"
Sep 15 17:10:43.999629 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:10:43.999608    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6\": container with ID starting with d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6 not found: ID does not exist" containerID="d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"
Sep 15 17:10:43.999673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999637    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6"} err="failed to get container status \"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6\": rpc error: code = NotFound desc = could not find container \"d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6\": container with ID starting with d7afe45ae71e09bfd91ebba9a7679f9e6a06d57ae56f64d99675ea2c95e06bb6 not found: ID does not exist"
Sep 15 17:10:43.999673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999653    2578 scope.go:117] "RemoveContainer" containerID="8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"
Sep 15 17:10:43.999929 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:10:43.999912    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626\": container with ID starting with 8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626 not found: ID does not exist" containerID="8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"
Sep 15 17:10:43.999990 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:43.999932    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626"} err="failed to get container status \"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626\": rpc error: code = NotFound desc = could not find container \"8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626\": container with ID starting with 8bdea5fe1b716e506f3d57c7bb38f0cd669f2e0d487dc95c99cf8a6c7e058626 not found: ID does not exist"
Sep 15 17:10:44.004767 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.004747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:10:44.010372 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.010351    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-ll5sb"]
Sep 15 17:10:44.978176 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.978134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerStarted","Data":"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"}
Sep 15 17:10:44.978609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.978199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerStarted","Data":"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"}
Sep 15 17:10:44.978609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.978232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:44.997466 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:44.997408    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podStartSLOduration=2.65525474 podStartE2EDuration="3.997392058s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.342137318s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:10:44.995143586 +0000 UTC m=+2616.381782880" watchObservedRunningTime="2026-09-15 17:10:44.997392058 +0000 UTC m=+2616.384031325"
Sep 15 17:10:45.154648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:45.154612    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" path="/var/lib/kubelet/pods/9efa8645-0553-4f6f-9d19-cf758e896a10/volumes"
Sep 15 17:10:45.988403 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:45.988367    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:45.989882 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:45.989852    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:10:46.994244 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:46.994198    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:10:51.998378 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:51.998347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:10:51.998907 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:10:51.998876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:01.999588 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:01.999548    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:11.999349 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:11.999306    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:21.999393 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:21.999354    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:31.999599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:31.999555    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:41.998839 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:41.998798    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:51.998922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:51.998881    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 15 17:11:57.154261 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:11:57.154229    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:12:01.295999 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:01.295915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:12:01.296401 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:01.296161    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container" containerID="cri-o://56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb" gracePeriod=30
Sep 15 17:12:01.296401 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:01.296207    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kube-rbac-proxy" containerID="cri-o://9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530" gracePeriod=30
Sep 15 17:12:01.995022 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:01.994963    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 15 17:12:02.338330 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.338236    2578 generic.go:356] "Generic (PLEG): container finished" podID="381b1006-cd1d-414f-8f8b-329c98959eba" containerID="9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530" exitCode=2
Sep 15 17:12:02.338330 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.338294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerDied","Data":"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"}
Sep 15 17:12:02.403984 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.403950    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:02.404316 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404304    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kube-rbac-proxy"
Sep 15 17:12:02.404362 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404317    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kube-rbac-proxy"
Sep 15 17:12:02.404362 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="storage-initializer"
Sep 15 17:12:02.404362 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404340    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="storage-initializer"
Sep 15 17:12:02.404362 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404347    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kserve-container"
Sep 15 17:12:02.404362 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404352    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kserve-container"
Sep 15 17:12:02.404519 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404405    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kube-rbac-proxy"
Sep 15 17:12:02.404519 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.404412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efa8645-0553-4f6f-9d19-cf758e896a10" containerName="kserve-container"
Sep 15 17:12:02.407486 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.407465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.410181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.410160    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 15 17:12:02.410289 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.410223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 15 17:12:02.419437 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.419418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:02.600617 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.600522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqbd\" (UniqueName: \"kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.600617 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.600566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.600823 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.600639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.600823 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.600682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702039 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702227 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702227 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqbd\" (UniqueName: \"kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702227 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702555 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.702781 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.702721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.704756 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.704719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.710673 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.710647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqbd\" (UniqueName: \"kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.718324 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.718303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:02.995384 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:02.995354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:02.997911 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:12:02.997873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb82aabc_283b_426c_90d0_f1c9c74d19d2.slice/crio-28ebb3fc227179ef981135976282aac6a8a7bd8bd7c52ce4239fde37131c48c0 WatchSource:0}: Error finding container 28ebb3fc227179ef981135976282aac6a8a7bd8bd7c52ce4239fde37131c48c0: Status 404 returned error can't find the container with id 28ebb3fc227179ef981135976282aac6a8a7bd8bd7c52ce4239fde37131c48c0
Sep 15 17:12:03.346856 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:03.346758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerStarted","Data":"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"}
Sep 15 17:12:03.346856 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:03.346828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerStarted","Data":"28ebb3fc227179ef981135976282aac6a8a7bd8bd7c52ce4239fde37131c48c0"}
Sep 15 17:12:05.847406 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:05.847384    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:12:06.032088 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.031985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config\") pod \"381b1006-cd1d-414f-8f8b-329c98959eba\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") "
Sep 15 17:12:06.032088 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032027    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls\") pod \"381b1006-cd1d-414f-8f8b-329c98959eba\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") "
Sep 15 17:12:06.032325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert\") pod \"381b1006-cd1d-414f-8f8b-329c98959eba\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") "
Sep 15 17:12:06.032325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location\") pod \"381b1006-cd1d-414f-8f8b-329c98959eba\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") "
Sep 15 17:12:06.032325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz962\" (UniqueName: \"kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962\") pod \"381b1006-cd1d-414f-8f8b-329c98959eba\" (UID: \"381b1006-cd1d-414f-8f8b-329c98959eba\") "
Sep 15 17:12:06.032468 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "381b1006-cd1d-414f-8f8b-329c98959eba" (UID: "381b1006-cd1d-414f-8f8b-329c98959eba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:12:06.032526 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "381b1006-cd1d-414f-8f8b-329c98959eba" (UID: "381b1006-cd1d-414f-8f8b-329c98959eba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:12:06.032569 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.032500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "381b1006-cd1d-414f-8f8b-329c98959eba" (UID: "381b1006-cd1d-414f-8f8b-329c98959eba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:12:06.034287 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.034267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "381b1006-cd1d-414f-8f8b-329c98959eba" (UID: "381b1006-cd1d-414f-8f8b-329c98959eba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:12:06.034493 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.034477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962" (OuterVolumeSpecName: "kube-api-access-dz962") pod "381b1006-cd1d-414f-8f8b-329c98959eba" (UID: "381b1006-cd1d-414f-8f8b-329c98959eba"). InnerVolumeSpecName "kube-api-access-dz962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:12:06.133682 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.133648    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:06.133682 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.133679    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/381b1006-cd1d-414f-8f8b-329c98959eba-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:06.133682 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.133690    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz962\" (UniqueName: \"kubernetes.io/projected/381b1006-cd1d-414f-8f8b-329c98959eba-kube-api-access-dz962\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:06.133938 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.133698    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/381b1006-cd1d-414f-8f8b-329c98959eba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:06.133938 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.133708    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/381b1006-cd1d-414f-8f8b-329c98959eba-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:06.363381 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.363299    2578 generic.go:356] "Generic (PLEG): container finished" podID="381b1006-cd1d-414f-8f8b-329c98959eba" containerID="56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb" exitCode=0
Sep 15 17:12:06.363381 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.363359    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"
Sep 15 17:12:06.363590 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.363376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerDied","Data":"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"}
Sep 15 17:12:06.363590 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.363437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559" event={"ID":"381b1006-cd1d-414f-8f8b-329c98959eba","Type":"ContainerDied","Data":"7c6bcf97402cb1be5292ac02b3e4160088f503f95c56b612b69b07fabb5222e5"}
Sep 15 17:12:06.363590 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.363463    2578 scope.go:117] "RemoveContainer" containerID="9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"
Sep 15 17:12:06.372399 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.372384    2578 scope.go:117] "RemoveContainer" containerID="56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"
Sep 15 17:12:06.379841 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.379804    2578 scope.go:117] "RemoveContainer" containerID="db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"
Sep 15 17:12:06.387177 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.387154    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:12:06.387552 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.387535    2578 scope.go:117] "RemoveContainer" containerID="9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"
Sep 15 17:12:06.387823 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:06.387806    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530\": container with ID starting with 9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530 not found: ID does not exist" containerID="9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"
Sep 15 17:12:06.387889 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.387832    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530"} err="failed to get container status \"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530\": rpc error: code = NotFound desc = could not find container \"9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530\": container with ID starting with 9b84c89b7495cf2c4280612ca00a566423395d79d8cbc541d296377e70dba530 not found: ID does not exist"
Sep 15 17:12:06.387889 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.387849    2578 scope.go:117] "RemoveContainer" containerID="56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"
Sep 15 17:12:06.388063 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:06.388046    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb\": container with ID starting with 56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb not found: ID does not exist" containerID="56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"
Sep 15 17:12:06.388102 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.388066    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb"} err="failed to get container status \"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb\": rpc error: code = NotFound desc = could not find container \"56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb\": container with ID starting with 56c039916f42416f04777a801de3bd18ae379c678f585b3d106c4b4f4e6e7cfb not found: ID does not exist"
Sep 15 17:12:06.388102 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.388079    2578 scope.go:117] "RemoveContainer" containerID="db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"
Sep 15 17:12:06.388363 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:06.388346    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1\": container with ID starting with db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1 not found: ID does not exist" containerID="db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"
Sep 15 17:12:06.388420 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.388370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1"} err="failed to get container status \"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1\": rpc error: code = NotFound desc = could not find container \"db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1\": container with ID starting with db0a143fcb7f0de55d0ee212b997acef367d7c13eedadf63cc5d6024806c2bd1 not found: ID does not exist"
Sep 15 17:12:06.391018 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:06.390997    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5c9ddf676d-qm559"]
Sep 15 17:12:07.154441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:07.154406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" path="/var/lib/kubelet/pods/381b1006-cd1d-414f-8f8b-329c98959eba/volumes"
Sep 15 17:12:08.375802 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:08.375774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:08.376165 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:08.375812    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerID="25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b" exitCode=1
Sep 15 17:12:08.376165 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:08.375849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerDied","Data":"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"}
Sep 15 17:12:09.382646 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:09.382613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:09.383074 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:09.382731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerStarted","Data":"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"}
Sep 15 17:12:11.757048 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:11.757023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:11.761506 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:11.761484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:12.382454 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.382419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:12.382767 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.382715    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer" containerID="cri-o://18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc" gracePeriod=30
Sep 15 17:12:12.777685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.777649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="storage-initializer"
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="storage-initializer"
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778047    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kube-rbac-proxy"
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778052    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kube-rbac-proxy"
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778062    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container"
Sep 15 17:12:12.778101 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778067    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container"
Sep 15 17:12:12.778333 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kube-rbac-proxy"
Sep 15 17:12:12.778333 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.778127    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="381b1006-cd1d-414f-8f8b-329c98959eba" containerName="kserve-container"
Sep 15 17:12:12.781323 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.781304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.783675 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.783654    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:12:12.783942 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.783923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 15 17:12:12.784007 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.783925    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:12:12.784183 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.784164    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 15 17:12:12.792079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.792057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:12:12.893706 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.893680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.893830 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.893721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps97l\" (UniqueName: \"kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.893878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.893820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.893927 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.893904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.893974 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.893932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994516 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994669 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994669 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994669 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994874 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps97l\" (UniqueName: \"kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.994874 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:12.994776    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 15 17:12:12.994874 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:12.994846    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls podName:66427b86-b945-4902-a871-ecd5952a218d nodeName:}" failed. No retries permitted until 2026-09-15 17:12:13.494826043 +0000 UTC m=+2704.881465288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-5f965" (UID: "66427b86-b945-4902-a871-ecd5952a218d") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 15 17:12:12.995044 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.994961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.995274 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.995251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:12.995361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:12.995345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:13.003693 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.003664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps97l\" (UniqueName: \"kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:13.029714 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.029667    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/1.log"
Sep 15 17:12:13.030028 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.030014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:13.033273 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.033258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:13.094944 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.094912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls\") pod \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") "
Sep 15 17:12:13.095102 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.094977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jqbd\" (UniqueName: \"kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd\") pod \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") "
Sep 15 17:12:13.095102 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.095072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config\") pod \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") "
Sep 15 17:12:13.095212 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.095116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location\") pod \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\" (UID: \"cb82aabc-283b-426c-90d0-f1c9c74d19d2\") "
Sep 15 17:12:13.095421 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.095395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb82aabc-283b-426c-90d0-f1c9c74d19d2" (UID: "cb82aabc-283b-426c-90d0-f1c9c74d19d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:12:13.095421 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.095405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb82aabc-283b-426c-90d0-f1c9c74d19d2" (UID: "cb82aabc-283b-426c-90d0-f1c9c74d19d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:12:13.097090 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.097070    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb82aabc-283b-426c-90d0-f1c9c74d19d2" (UID: "cb82aabc-283b-426c-90d0-f1c9c74d19d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:12:13.097152 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.097134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd" (OuterVolumeSpecName: "kube-api-access-8jqbd") pod "cb82aabc-283b-426c-90d0-f1c9c74d19d2" (UID: "cb82aabc-283b-426c-90d0-f1c9c74d19d2"). InnerVolumeSpecName "kube-api-access-8jqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:12:13.196907 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.196875    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jqbd\" (UniqueName: \"kubernetes.io/projected/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-api-access-8jqbd\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:13.196907 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.196910    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:13.197130 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.196927    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb82aabc-283b-426c-90d0-f1c9c74d19d2-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:13.197130 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.196948    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb82aabc-283b-426c-90d0-f1c9c74d19d2-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:12:13.402695 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.402665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/1.log"
Sep 15 17:12:13.403076 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403062    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n_cb82aabc-283b-426c-90d0-f1c9c74d19d2/storage-initializer/0.log"
Sep 15 17:12:13.403150 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403096    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerID="18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc" exitCode=1
Sep 15 17:12:13.403150 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerDied","Data":"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"}
Sep 15 17:12:13.403265 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403192    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"
Sep 15 17:12:13.403265 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n" event={"ID":"cb82aabc-283b-426c-90d0-f1c9c74d19d2","Type":"ContainerDied","Data":"28ebb3fc227179ef981135976282aac6a8a7bd8bd7c52ce4239fde37131c48c0"}
Sep 15 17:12:13.403265 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.403238    2578 scope.go:117] "RemoveContainer" containerID="18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"
Sep 15 17:12:13.412306 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.412282    2578 scope.go:117] "RemoveContainer" containerID="25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"
Sep 15 17:12:13.419954 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.419938    2578 scope.go:117] "RemoveContainer" containerID="18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"
Sep 15 17:12:13.420199 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:13.420181    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc\": container with ID starting with 18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc not found: ID does not exist" containerID="18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"
Sep 15 17:12:13.420258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.420207    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc"} err="failed to get container status \"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc\": rpc error: code = NotFound desc = could not find container \"18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc\": container with ID starting with 18e1934cf0733e7d66dd9ecf446f2f1f9e721bc61f79269fdf11e44a7de0b5bc not found: ID does not exist"
Sep 15 17:12:13.420258 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.420222    2578 scope.go:117] "RemoveContainer" containerID="25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"
Sep 15 17:12:13.420467 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:12:13.420454    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b\": container with ID starting with 25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b not found: ID does not exist" containerID="25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"
Sep 15 17:12:13.420509 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.420469    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b"} err="failed to get container status \"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b\": rpc error: code = NotFound desc = could not find container \"25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b\": container with ID starting with 25b71b9ae931d89aeafe1956a783d805d0ac880227bb66a90d3d4ef56012ec4b not found: ID does not exist"
Sep 15 17:12:13.437535 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.437508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:13.445594 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.445563    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6f774b8cd8-29z9n"]
Sep 15 17:12:13.487041 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487011    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:12:13.487371 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487360    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.487412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487373    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.487445 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487440    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.487476 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487449    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.487511 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487494    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.487511 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.487501    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" containerName="storage-initializer"
Sep 15 17:12:13.491971 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.491950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.494171 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.494150    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 15 17:12:13.494308 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.494283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:12:13.494423 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.494406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 15 17:12:13.499034 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.498969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:13.502200 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.502176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5f965\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:13.502292 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.502274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:12:13.600006 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.599969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.600196 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.600075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.600196 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.600160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdcj\" (UniqueName: \"kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.600196 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.600195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.600346 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.600226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.692371 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.692279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:13.701479 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.701585 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdcj\" (UniqueName: \"kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.701585 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.701699 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.701699 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.701873 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.701848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.702226 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.702205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.702325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.702295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.704086 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.704066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.710626 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.710600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdcj\" (UniqueName: \"kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.802927 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.802890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:13.964460 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:13.964428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:12:13.966219 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:12:13.966193    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66427b86_b945_4902_a871_ecd5952a218d.slice/crio-e80753b3f54d78dfa467b5d46488c0831fecdb497814c500adabfe3547144809 WatchSource:0}: Error finding container e80753b3f54d78dfa467b5d46488c0831fecdb497814c500adabfe3547144809: Status 404 returned error can't find the container with id e80753b3f54d78dfa467b5d46488c0831fecdb497814c500adabfe3547144809
Sep 15 17:12:14.072439 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:14.072416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:12:14.074586 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:12:14.074558    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fe680e_8092_4a13_836f_a306a7e31c6e.slice/crio-4d501c7e7a2d887ad91e1d62eeabbc1909868299eda77c201b92f670d42289c8 WatchSource:0}: Error finding container 4d501c7e7a2d887ad91e1d62eeabbc1909868299eda77c201b92f670d42289c8: Status 404 returned error can't find the container with id 4d501c7e7a2d887ad91e1d62eeabbc1909868299eda77c201b92f670d42289c8
Sep 15 17:12:14.409906 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:14.409865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerStarted","Data":"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"}
Sep 15 17:12:14.410123 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:14.409953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerStarted","Data":"e80753b3f54d78dfa467b5d46488c0831fecdb497814c500adabfe3547144809"}
Sep 15 17:12:14.411831 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:14.411803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerStarted","Data":"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"}
Sep 15 17:12:14.411965 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:14.411859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerStarted","Data":"4d501c7e7a2d887ad91e1d62eeabbc1909868299eda77c201b92f670d42289c8"}
Sep 15 17:12:15.154679 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:15.154649    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb82aabc-283b-426c-90d0-f1c9c74d19d2" path="/var/lib/kubelet/pods/cb82aabc-283b-426c-90d0-f1c9c74d19d2/volumes"
Sep 15 17:12:15.419257 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:15.419167    2578 generic.go:356] "Generic (PLEG): container finished" podID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerID="c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133" exitCode=0
Sep 15 17:12:15.419406 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:15.419257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerDied","Data":"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"}
Sep 15 17:12:16.427025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:16.426986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerStarted","Data":"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"}
Sep 15 17:12:16.427418 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:16.427046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerStarted","Data":"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"}
Sep 15 17:12:16.427418 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:16.427079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:16.446522 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:16.446474    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podStartSLOduration=2.325435665 podStartE2EDuration="3.44645647s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.121020805s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:12:16.444231345 +0000 UTC m=+2707.830870622" watchObservedRunningTime="2026-09-15 17:12:16.44645647 +0000 UTC m=+2707.833095737"
Sep 15 17:12:17.432181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:17.432145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:17.433402 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:17.433362    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:18.436985 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:18.436939    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:19.442482 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:19.442446    2578 generic.go:356] "Generic (PLEG): container finished" podID="66427b86-b945-4902-a871-ecd5952a218d" containerID="aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067" exitCode=0
Sep 15 17:12:19.442888 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:19.442521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerDied","Data":"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"}
Sep 15 17:12:19.443861 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:19.443844    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 17:12:23.442548 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:23.442520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:12:23.443235 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:23.443204    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:31.517024 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.516991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerStarted","Data":"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"}
Sep 15 17:12:31.517434 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.517052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerStarted","Data":"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"}
Sep 15 17:12:31.517434 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.517312    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:31.517551 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.517469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:31.518599 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.518577    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:12:31.536800 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:31.536706    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podStartSLOduration=2.837737961 podStartE2EDuration="19.536691326s" totalImagesPullingTime="11.910189165s" totalInitContainerRuntime="4.7887642s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:12:31.535941861 +0000 UTC m=+2722.922581127" watchObservedRunningTime="2026-09-15 17:12:31.536691326 +0000 UTC m=+2722.923330594"
Sep 15 17:12:32.521975 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:32.521940    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:12:33.443943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:33.443897    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:37.526156 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:37.526127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:12:37.526627 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:37.526601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:12:43.443871 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:43.443833    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:47.527302 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:47.527262    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:12:53.443420 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:53.443373    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:12:57.526902 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:12:57.526864    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:13:03.443069 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:03.443028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:13:07.527601 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:07.527559    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 15 17:13:13.443800 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:13.443760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:13:17.527490 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:17.527460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:13:23.443952 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:23.443907    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 15 17:13:23.918816 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:23.918781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:13:23.919113 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:23.919074    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container" containerID="cri-o://1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a" gracePeriod=30
Sep 15 17:13:23.919191 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:23.919141    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kube-rbac-proxy" containerID="cri-o://d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9" gracePeriod=30
Sep 15 17:13:24.773380 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:24.773335    2578 generic.go:356] "Generic (PLEG): container finished" podID="66427b86-b945-4902-a871-ecd5952a218d" containerID="d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9" exitCode=2
Sep 15 17:13:24.773774 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:24.773405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerDied","Data":"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"}
Sep 15 17:13:26.557486 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.557461    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:13:26.653624 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.653545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location\") pod \"66427b86-b945-4902-a871-ecd5952a218d\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") "
Sep 15 17:13:26.653798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.653631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert\") pod \"66427b86-b945-4902-a871-ecd5952a218d\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") "
Sep 15 17:13:26.653798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.653658    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") pod \"66427b86-b945-4902-a871-ecd5952a218d\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") "
Sep 15 17:13:26.653798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.653675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps97l\" (UniqueName: \"kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l\") pod \"66427b86-b945-4902-a871-ecd5952a218d\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") "
Sep 15 17:13:26.653798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.653698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config\") pod \"66427b86-b945-4902-a871-ecd5952a218d\" (UID: \"66427b86-b945-4902-a871-ecd5952a218d\") "
Sep 15 17:13:26.654069 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.654050    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66427b86-b945-4902-a871-ecd5952a218d" (UID: "66427b86-b945-4902-a871-ecd5952a218d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:13:26.654139 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.654112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66427b86-b945-4902-a871-ecd5952a218d" (UID: "66427b86-b945-4902-a871-ecd5952a218d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:13:26.655929 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.655905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66427b86-b945-4902-a871-ecd5952a218d" (UID: "66427b86-b945-4902-a871-ecd5952a218d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:13:26.656038 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.656016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l" (OuterVolumeSpecName: "kube-api-access-ps97l") pod "66427b86-b945-4902-a871-ecd5952a218d" (UID: "66427b86-b945-4902-a871-ecd5952a218d"). InnerVolumeSpecName "kube-api-access-ps97l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:13:26.663608 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.663581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66427b86-b945-4902-a871-ecd5952a218d" (UID: "66427b86-b945-4902-a871-ecd5952a218d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:13:26.754957 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.754921    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66427b86-b945-4902-a871-ecd5952a218d-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:26.754957 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.754952    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:26.754957 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.754961    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66427b86-b945-4902-a871-ecd5952a218d-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:26.755157 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.754969    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps97l\" (UniqueName: \"kubernetes.io/projected/66427b86-b945-4902-a871-ecd5952a218d-kube-api-access-ps97l\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:26.755157 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.754978    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66427b86-b945-4902-a871-ecd5952a218d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:26.783797 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.783761    2578 generic.go:356] "Generic (PLEG): container finished" podID="66427b86-b945-4902-a871-ecd5952a218d" containerID="1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a" exitCode=0
Sep 15 17:13:26.783943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.783803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerDied","Data":"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"}
Sep 15 17:13:26.783943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.783823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"
Sep 15 17:13:26.783943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.783904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965" event={"ID":"66427b86-b945-4902-a871-ecd5952a218d","Type":"ContainerDied","Data":"e80753b3f54d78dfa467b5d46488c0831fecdb497814c500adabfe3547144809"}
Sep 15 17:13:26.784119 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.783949    2578 scope.go:117] "RemoveContainer" containerID="d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"
Sep 15 17:13:26.793413 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.793396    2578 scope.go:117] "RemoveContainer" containerID="1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"
Sep 15 17:13:26.801340 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.801319    2578 scope.go:117] "RemoveContainer" containerID="aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"
Sep 15 17:13:26.805846 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.805816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:13:26.810132 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.810110    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5f965"]
Sep 15 17:13:26.810776 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.810760    2578 scope.go:117] "RemoveContainer" containerID="d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"
Sep 15 17:13:26.811049 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:26.811028    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9\": container with ID starting with d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9 not found: ID does not exist" containerID="d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"
Sep 15 17:13:26.811112 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.811055    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9"} err="failed to get container status \"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9\": rpc error: code = NotFound desc = could not find container \"d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9\": container with ID starting with d84a52b265aede3526b5b1c3a7dfe607822e05c199e1421c4d19dcdaa9c5c8c9 not found: ID does not exist"
Sep 15 17:13:26.811112 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.811072    2578 scope.go:117] "RemoveContainer" containerID="1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"
Sep 15 17:13:26.811333 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:26.811317    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a\": container with ID starting with 1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a not found: ID does not exist" containerID="1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"
Sep 15 17:13:26.811375 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.811337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a"} err="failed to get container status \"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a\": rpc error: code = NotFound desc = could not find container \"1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a\": container with ID starting with 1daf87bfd828710b1d035184c6640795cbe3928e6d0e086163291a7f888a167a not found: ID does not exist"
Sep 15 17:13:26.811375 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.811350    2578 scope.go:117] "RemoveContainer" containerID="aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"
Sep 15 17:13:26.811605 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:26.811586    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067\": container with ID starting with aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067 not found: ID does not exist" containerID="aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"
Sep 15 17:13:26.811656 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:26.811614    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067"} err="failed to get container status \"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067\": rpc error: code = NotFound desc = could not find container \"aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067\": container with ID starting with aa07e1f19ca1a4271e4df38056ddbaccc7cb50aff63751e95b677642177b8067 not found: ID does not exist"
Sep 15 17:13:27.154392 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:27.154358    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66427b86-b945-4902-a871-ecd5952a218d" path="/var/lib/kubelet/pods/66427b86-b945-4902-a871-ecd5952a218d/volumes"
Sep 15 17:13:29.127857 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.127820    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128164    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="storage-initializer"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128175    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="storage-initializer"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128184    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kube-rbac-proxy"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kube-rbac-proxy"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128267    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kserve-container"
Sep 15 17:13:29.128348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.128277    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="66427b86-b945-4902-a871-ecd5952a218d" containerName="kube-rbac-proxy"
Sep 15 17:13:29.131851 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.131828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.134079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.134057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 15 17:13:29.134171 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.134129    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:13:29.134354 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.134338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:13:29.142678 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.142656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:13:29.276934 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.276903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.277121 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.276981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwx5r\" (UniqueName: \"kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.277121 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.277007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.277121 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.277027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.277121 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.277053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378078 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.377992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378078 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378303 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378303 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwx5r\" (UniqueName: \"kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378303 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378493 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.378980 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.378893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.380941 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.380921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.386318 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.386291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwx5r\" (UniqueName: \"kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r\") pod \"isvc-paddle-runtime-predictor-58b99fff76-5lckb\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.443091 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.443061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:29.718988 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.718913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:13:29.721349 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:13:29.721320    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85afdcc_af4e_48e6_b9b2_f75776739396.slice/crio-241681161fa6cbb338a27e513c5f1b23247493798445be08651b7534f319c945 WatchSource:0}: Error finding container 241681161fa6cbb338a27e513c5f1b23247493798445be08651b7534f319c945: Status 404 returned error can't find the container with id 241681161fa6cbb338a27e513c5f1b23247493798445be08651b7534f319c945
Sep 15 17:13:29.800129 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.800041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerStarted","Data":"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"}
Sep 15 17:13:29.800279 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:29.800159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerStarted","Data":"241681161fa6cbb338a27e513c5f1b23247493798445be08651b7534f319c945"}
Sep 15 17:13:33.443588 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:33.443511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:13:34.825999 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:34.825965    2578 generic.go:356] "Generic (PLEG): container finished" podID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerID="56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7" exitCode=0
Sep 15 17:13:34.826383 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:34.826022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerDied","Data":"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"}
Sep 15 17:13:35.832778 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:35.832721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerStarted","Data":"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"}
Sep 15 17:13:35.833178 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:35.832813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerStarted","Data":"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"}
Sep 15 17:13:35.833178 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:35.833026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:36.837357 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:36.837323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:36.838524 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:36.838494    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:13:37.842163 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:37.842114    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:13:42.846259 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:42.846226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:13:42.846763 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:42.846691    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:13:42.866977 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:42.866923    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podStartSLOduration=9.494734672 podStartE2EDuration="13.866907686s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.372173014s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:13:35.852294103 +0000 UTC m=+2787.238933370" watchObservedRunningTime="2026-09-15 17:13:42.866907686 +0000 UTC m=+2794.253546953"
Sep 15 17:13:43.505841 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:43.505808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:13:43.506144 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:43.506116    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container" containerID="cri-o://76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37" gracePeriod=30
Sep 15 17:13:43.506201 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:43.506176    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kube-rbac-proxy" containerID="cri-o://e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59" gracePeriod=30
Sep 15 17:13:43.872970 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:43.872891    2578 generic.go:356] "Generic (PLEG): container finished" podID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerID="e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59" exitCode=2
Sep 15 17:13:43.872970 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:43.872945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerDied","Data":"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"}
Sep 15 17:13:48.245948 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.245926    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:13:48.343747 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.343715    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location\") pod \"40fe680e-8092-4a13-836f-a306a7e31c6e\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") "
Sep 15 17:13:48.343926 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.343770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert\") pod \"40fe680e-8092-4a13-836f-a306a7e31c6e\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") "
Sep 15 17:13:48.343926 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.343788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config\") pod \"40fe680e-8092-4a13-836f-a306a7e31c6e\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") "
Sep 15 17:13:48.343926 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.343835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhdcj\" (UniqueName: \"kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj\") pod \"40fe680e-8092-4a13-836f-a306a7e31c6e\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") "
Sep 15 17:13:48.343926 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.343853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls\") pod \"40fe680e-8092-4a13-836f-a306a7e31c6e\" (UID: \"40fe680e-8092-4a13-836f-a306a7e31c6e\") "
Sep 15 17:13:48.344154 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.344090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40fe680e-8092-4a13-836f-a306a7e31c6e" (UID: "40fe680e-8092-4a13-836f-a306a7e31c6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:13:48.344154 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.344141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40fe680e-8092-4a13-836f-a306a7e31c6e" (UID: "40fe680e-8092-4a13-836f-a306a7e31c6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:13:48.344235 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.344153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40fe680e-8092-4a13-836f-a306a7e31c6e" (UID: "40fe680e-8092-4a13-836f-a306a7e31c6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:13:48.346093 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.346044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40fe680e-8092-4a13-836f-a306a7e31c6e" (UID: "40fe680e-8092-4a13-836f-a306a7e31c6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:13:48.346181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.346157    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj" (OuterVolumeSpecName: "kube-api-access-qhdcj") pod "40fe680e-8092-4a13-836f-a306a7e31c6e" (UID: "40fe680e-8092-4a13-836f-a306a7e31c6e"). InnerVolumeSpecName "kube-api-access-qhdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:13:48.444798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.444763    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40fe680e-8092-4a13-836f-a306a7e31c6e-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:48.444798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.444791    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:48.444798 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.444801    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:48.445030 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.444809    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdcj\" (UniqueName: \"kubernetes.io/projected/40fe680e-8092-4a13-836f-a306a7e31c6e-kube-api-access-qhdcj\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:48.445030 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.444819    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40fe680e-8092-4a13-836f-a306a7e31c6e-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:13:48.901380 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.901337    2578 generic.go:356] "Generic (PLEG): container finished" podID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerID="76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37" exitCode=0
Sep 15 17:13:48.901561 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.901478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerDied","Data":"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"}
Sep 15 17:13:48.901561 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.901491    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"
Sep 15 17:13:48.901647 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.901567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp" event={"ID":"40fe680e-8092-4a13-836f-a306a7e31c6e","Type":"ContainerDied","Data":"4d501c7e7a2d887ad91e1d62eeabbc1909868299eda77c201b92f670d42289c8"}
Sep 15 17:13:48.901647 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.901597    2578 scope.go:117] "RemoveContainer" containerID="e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"
Sep 15 17:13:48.910400 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.910375    2578 scope.go:117] "RemoveContainer" containerID="76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"
Sep 15 17:13:48.920171 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.920146    2578 scope.go:117] "RemoveContainer" containerID="c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"
Sep 15 17:13:48.926177 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.926150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:13:48.928250 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.928226    2578 scope.go:117] "RemoveContainer" containerID="e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"
Sep 15 17:13:48.928543 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:48.928518    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59\": container with ID starting with e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59 not found: ID does not exist" containerID="e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"
Sep 15 17:13:48.928633 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.928550    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59"} err="failed to get container status \"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59\": rpc error: code = NotFound desc = could not find container \"e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59\": container with ID starting with e500c0e4349a70886ac7d1deee3a62c55b87e12968e4c58a2e2513e91018ad59 not found: ID does not exist"
Sep 15 17:13:48.928633 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.928569    2578 scope.go:117] "RemoveContainer" containerID="76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"
Sep 15 17:13:48.928943 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:48.928914    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37\": container with ID starting with 76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37 not found: ID does not exist" containerID="76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"
Sep 15 17:13:48.929040 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.928952    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37"} err="failed to get container status \"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37\": rpc error: code = NotFound desc = could not find container \"76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37\": container with ID starting with 76714ae7a9cf3ba64f203bd10f8eac2158817aa7a5963c69d9634455529c1b37 not found: ID does not exist"
Sep 15 17:13:48.929040 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.928976    2578 scope.go:117] "RemoveContainer" containerID="c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"
Sep 15 17:13:48.929499 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:48.929469    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133\": container with ID starting with c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133 not found: ID does not exist" containerID="c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"
Sep 15 17:13:48.929659 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.929508    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133"} err="failed to get container status \"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133\": rpc error: code = NotFound desc = could not find container \"c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133\": container with ID starting with c44840a03a4988bf004e3bddc2e6c0405cce201c4ffbb739fdd2bda184398133 not found: ID does not exist"
Sep 15 17:13:48.931569 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:48.931549    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85bbbdf56d-qfsrp"]
Sep 15 17:13:49.154760 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:49.154673    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" path="/var/lib/kubelet/pods/40fe680e-8092-4a13-836f-a306a7e31c6e/volumes"
Sep 15 17:13:52.847433 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:52.847392    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:13:55.725965 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.725927    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:13:55.726378 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726310    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kube-rbac-proxy"
Sep 15 17:13:55.726378 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726321    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kube-rbac-proxy"
Sep 15 17:13:55.726378 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726369    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="storage-initializer"
Sep 15 17:13:55.726541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726409    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="storage-initializer"
Sep 15 17:13:55.726541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726416    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container"
Sep 15 17:13:55.726541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726423    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container"
Sep 15 17:13:55.726541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726485    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kube-rbac-proxy"
Sep 15 17:13:55.726541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.726493    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fe680e-8092-4a13-836f-a306a7e31c6e" containerName="kserve-container"
Sep 15 17:13:55.729107 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.729090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.731275 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.731253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 15 17:13:55.731415 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.731373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 15 17:13:55.731641 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.731626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:13:55.740547 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.740523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:13:55.813771 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.813722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.813932 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.813790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.813932 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.813881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fmn\" (UniqueName: \"kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.814011 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.813931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.814011 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.813986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.914859 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.914817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.914995 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.914879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.914995 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.914931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fmn\" (UniqueName: \"kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.914995 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.914967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.915181 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.915010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.915181 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:55.915131    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 15 17:13:55.915285 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:13:55.915217    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls podName:213218d9-bbff-4fed-96d2-d9cf7ce2cd64 nodeName:}" failed. No retries permitted until 2026-09-15 17:13:56.415192759 +0000 UTC m=+2807.801832023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 15 17:13:55.915426 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.915402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.915541 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.915523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.915583 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.915557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:55.924129 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:55.924111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fmn\" (UniqueName: \"kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:56.419607 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:56.419568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:56.422064 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:56.422042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:56.639900 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:56.639850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:56.914480 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:56.914447    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:13:56.917850 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:13:56.917823    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213218d9_bbff_4fed_96d2_d9cf7ce2cd64.slice/crio-abe4824707b73a316934a0f34be28fb3d3b99c0194e0c2d06355ccc7232606d3 WatchSource:0}: Error finding container abe4824707b73a316934a0f34be28fb3d3b99c0194e0c2d06355ccc7232606d3: Status 404 returned error can't find the container with id abe4824707b73a316934a0f34be28fb3d3b99c0194e0c2d06355ccc7232606d3
Sep 15 17:13:56.940412 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:56.940355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerStarted","Data":"abe4824707b73a316934a0f34be28fb3d3b99c0194e0c2d06355ccc7232606d3"}
Sep 15 17:13:57.948873 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:57.948839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerStarted","Data":"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"}
Sep 15 17:13:58.954878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:58.954844    2578 generic.go:356] "Generic (PLEG): container finished" podID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerID="a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41" exitCode=0
Sep 15 17:13:58.955331 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:58.954933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerDied","Data":"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"}
Sep 15 17:13:59.961610 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.961575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerStarted","Data":"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"}
Sep 15 17:13:59.962028 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.961636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerStarted","Data":"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"}
Sep 15 17:13:59.962028 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.961668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:59.962028 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.961689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:13:59.963093 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.963069    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:13:59.982904 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:13:59.982866    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podStartSLOduration=3.862998223 podStartE2EDuration="4.982852925s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.119854702s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:13:59.980237282 +0000 UTC m=+2811.366876563" watchObservedRunningTime="2026-09-15 17:13:59.982852925 +0000 UTC m=+2811.369492193"
Sep 15 17:14:00.966128 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:00.966091    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:02.847547 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:02.847504    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:14:05.971031 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:05.970998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:14:05.971470 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:05.971446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:12.847605 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:12.847562    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:14:15.972319 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:15.972276    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:22.847532 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:22.847500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:14:25.971531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:25.971495    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:30.216021 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:30.215986    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:14:30.216719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:30.216275    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" containerID="cri-o://fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce" gracePeriod=30
Sep 15 17:14:30.216719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:30.216311    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kube-rbac-proxy" containerID="cri-o://611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8" gracePeriod=30
Sep 15 17:14:31.107719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:31.107687    2578 generic.go:356] "Generic (PLEG): container finished" podID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerID="611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8" exitCode=2
Sep 15 17:14:31.107909 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:31.107759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerDied","Data":"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"}
Sep 15 17:14:32.842391 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:32.842356    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 15 17:14:32.846716 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:32.846684    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 15 17:14:32.953894 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:32.953870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:14:33.042721 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.042679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls\") pod \"b85afdcc-af4e-48e6-b9b2-f75776739396\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") "
Sep 15 17:14:33.042922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.042765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config\") pod \"b85afdcc-af4e-48e6-b9b2-f75776739396\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") "
Sep 15 17:14:33.042922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.042798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location\") pod \"b85afdcc-af4e-48e6-b9b2-f75776739396\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") "
Sep 15 17:14:33.042922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.042868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwx5r\" (UniqueName: \"kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r\") pod \"b85afdcc-af4e-48e6-b9b2-f75776739396\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") "
Sep 15 17:14:33.042922 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.042890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert\") pod \"b85afdcc-af4e-48e6-b9b2-f75776739396\" (UID: \"b85afdcc-af4e-48e6-b9b2-f75776739396\") "
Sep 15 17:14:33.043262 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.043225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b85afdcc-af4e-48e6-b9b2-f75776739396" (UID: "b85afdcc-af4e-48e6-b9b2-f75776739396"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:14:33.043392 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.043276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b85afdcc-af4e-48e6-b9b2-f75776739396" (UID: "b85afdcc-af4e-48e6-b9b2-f75776739396"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:14:33.044925 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.044899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b85afdcc-af4e-48e6-b9b2-f75776739396" (UID: "b85afdcc-af4e-48e6-b9b2-f75776739396"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:14:33.045073 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.045056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r" (OuterVolumeSpecName: "kube-api-access-cwx5r") pod "b85afdcc-af4e-48e6-b9b2-f75776739396" (UID: "b85afdcc-af4e-48e6-b9b2-f75776739396"). InnerVolumeSpecName "kube-api-access-cwx5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:14:33.053253 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.053182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b85afdcc-af4e-48e6-b9b2-f75776739396" (UID: "b85afdcc-af4e-48e6-b9b2-f75776739396"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:14:33.119973 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.119939    2578 generic.go:356] "Generic (PLEG): container finished" podID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerID="fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce" exitCode=0
Sep 15 17:14:33.120143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.119996    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"
Sep 15 17:14:33.120143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.120001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerDied","Data":"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"}
Sep 15 17:14:33.120143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.120050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb" event={"ID":"b85afdcc-af4e-48e6-b9b2-f75776739396","Type":"ContainerDied","Data":"241681161fa6cbb338a27e513c5f1b23247493798445be08651b7534f319c945"}
Sep 15 17:14:33.120143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.120074    2578 scope.go:117] "RemoveContainer" containerID="611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"
Sep 15 17:14:33.128824 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.128805    2578 scope.go:117] "RemoveContainer" containerID="fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"
Sep 15 17:14:33.136608 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.136587    2578 scope.go:117] "RemoveContainer" containerID="56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"
Sep 15 17:14:33.141316 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.141285    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:14:33.143754 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.143704    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwx5r\" (UniqueName: \"kubernetes.io/projected/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-api-access-cwx5r\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:14:33.143754 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.143728    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:14:33.143923 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.143766    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b85afdcc-af4e-48e6-b9b2-f75776739396-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:14:33.143923 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.143774    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b85afdcc-af4e-48e6-b9b2-f75776739396-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:14:33.143923 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.143782    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85afdcc-af4e-48e6-b9b2-f75776739396-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:14:33.144965 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.144946    2578 scope.go:117] "RemoveContainer" containerID="611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"
Sep 15 17:14:33.145284 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:14:33.145265    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8\": container with ID starting with 611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8 not found: ID does not exist" containerID="611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"
Sep 15 17:14:33.145341 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145295    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8"} err="failed to get container status \"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8\": rpc error: code = NotFound desc = could not find container \"611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8\": container with ID starting with 611d846315531b279bc0174f76b3282a56b8cf69bb62446269bd190c1e1772e8 not found: ID does not exist"
Sep 15 17:14:33.145341 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145310    2578 scope.go:117] "RemoveContainer" containerID="fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"
Sep 15 17:14:33.145515 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145494    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-5lckb"]
Sep 15 17:14:33.145584 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:14:33.145552    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce\": container with ID starting with fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce not found: ID does not exist" containerID="fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"
Sep 15 17:14:33.145640 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145581    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce"} err="failed to get container status \"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce\": rpc error: code = NotFound desc = could not find container \"fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce\": container with ID starting with fdfd4b517273658fdb74831d9d5cd65329ae86fe60192c7f1c1214aebee587ce not found: ID does not exist"
Sep 15 17:14:33.145640 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145605    2578 scope.go:117] "RemoveContainer" containerID="56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"
Sep 15 17:14:33.145903 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:14:33.145889    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7\": container with ID starting with 56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7 not found: ID does not exist" containerID="56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"
Sep 15 17:14:33.145962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.145913    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7"} err="failed to get container status \"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7\": rpc error: code = NotFound desc = could not find container \"56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7\": container with ID starting with 56f43f3fdf42c3afd4fa954908f17649dcbfe38b0398988284cff99f0d98f2f7 not found: ID does not exist"
Sep 15 17:14:33.154620 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:33.154600    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" path="/var/lib/kubelet/pods/b85afdcc-af4e-48e6-b9b2-f75776739396/volumes"
Sep 15 17:14:35.407295 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:14:35.407755 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407717    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kube-rbac-proxy"
Sep 15 17:14:35.407755 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407749    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kube-rbac-proxy"
Sep 15 17:14:35.407836 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407760    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="storage-initializer"
Sep 15 17:14:35.407836 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407766    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="storage-initializer"
Sep 15 17:14:35.407836 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407797    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container"
Sep 15 17:14:35.407836 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container"
Sep 15 17:14:35.407967 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407864    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kserve-container"
Sep 15 17:14:35.407967 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.407873    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85afdcc-af4e-48e6-b9b2-f75776739396" containerName="kube-rbac-proxy"
Sep 15 17:14:35.412586 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.412566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.415069 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.415042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 15 17:14:35.415069 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.415060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 15 17:14:35.415243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.415087    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:14:35.420690 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.420666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:14:35.464934 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.464900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.465092 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.464965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pws2m\" (UniqueName: \"kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.465092 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.464992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.465092 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.465032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.465092 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.465080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566062 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pws2m\" (UniqueName: \"kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566062 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566359 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566359 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:14:35.566270    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 15 17:14:35.566359 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:14:35.566352    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls podName:3c421e01-c1dd-48ae-a2af-4827e1e5719e nodeName:}" failed. No retries permitted until 2026-09-15 17:14:36.066333024 +0000 UTC m=+2847.452972269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 15 17:14:35.566621 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566892 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.566935 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.566902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.575705 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.575679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pws2m\" (UniqueName: \"kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:35.972117 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:35.972081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:36.070034 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:36.069998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:36.072639 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:36.072609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:36.325202 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:36.325123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:36.602187 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:36.602160    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:14:36.604268 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:14:36.604239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c421e01_c1dd_48ae_a2af_4827e1e5719e.slice/crio-01af64f9eb413a8c04fc980be7dc3465c8f904f3d5a055b6fcc3bcc01b94094b WatchSource:0}: Error finding container 01af64f9eb413a8c04fc980be7dc3465c8f904f3d5a055b6fcc3bcc01b94094b: Status 404 returned error can't find the container with id 01af64f9eb413a8c04fc980be7dc3465c8f904f3d5a055b6fcc3bcc01b94094b
Sep 15 17:14:37.141243 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:37.141207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerStarted","Data":"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"}
Sep 15 17:14:37.141446 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:37.141269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerStarted","Data":"01af64f9eb413a8c04fc980be7dc3465c8f904f3d5a055b6fcc3bcc01b94094b"}
Sep 15 17:14:41.167943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:41.167910    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerID="c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e" exitCode=0
Sep 15 17:14:41.168302 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:41.167987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerDied","Data":"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"}
Sep 15 17:14:42.175630 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.175595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerStarted","Data":"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"}
Sep 15 17:14:42.176097 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.175653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerStarted","Data":"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"}
Sep 15 17:14:42.176097 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.175856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:42.176097 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.175902    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:42.177202 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.177178    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:14:42.195324 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:42.195280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podStartSLOduration=2.846075085 podStartE2EDuration="7.195265806s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.349190721s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:14:42.19279286 +0000 UTC m=+2853.579432128" watchObservedRunningTime="2026-09-15 17:14:42.195265806 +0000 UTC m=+2853.581905072"
Sep 15 17:14:43.180461 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:43.180422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:14:45.971699 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:45.971651    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:48.184709 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:48.184673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:14:48.185559 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:48.185271    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:14:55.971597 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:55.971552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:14:58.186002 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:14:58.185962    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:15:05.972091 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:05.971994    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:15:08.186286 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:08.186238    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:15:12.150009 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:12.149966    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 15 17:15:18.186068 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:18.186030    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:15:22.150875 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:22.150839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:15:25.728598 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:25.728561    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:15:25.729037 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:25.728817    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container" containerID="cri-o://77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4" gracePeriod=30
Sep 15 17:15:25.729037 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:25.728850    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kube-rbac-proxy" containerID="cri-o://d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331" gracePeriod=30
Sep 15 17:15:25.967050 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:25.967006    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 15 17:15:26.396665 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:26.396633    2578 generic.go:356] "Generic (PLEG): container finished" podID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerID="d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331" exitCode=2
Sep 15 17:15:26.396882 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:26.396677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerDied","Data":"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"}
Sep 15 17:15:28.186693 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:28.186658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:15:30.269182 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.269155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:15:30.314005 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.313973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") pod \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") "
Sep 15 17:15:30.314155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25fmn\" (UniqueName: \"kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn\") pod \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") "
Sep 15 17:15:30.314155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config\") pod \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") "
Sep 15 17:15:30.314155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location\") pod \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") "
Sep 15 17:15:30.314325 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert\") pod \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\" (UID: \"213218d9-bbff-4fed-96d2-d9cf7ce2cd64\") "
Sep 15 17:15:30.314495 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "213218d9-bbff-4fed-96d2-d9cf7ce2cd64" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:15:30.314616 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "213218d9-bbff-4fed-96d2-d9cf7ce2cd64" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:15:30.314616 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.314536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "213218d9-bbff-4fed-96d2-d9cf7ce2cd64" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:15:30.316285 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.316265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn" (OuterVolumeSpecName: "kube-api-access-25fmn") pod "213218d9-bbff-4fed-96d2-d9cf7ce2cd64" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64"). InnerVolumeSpecName "kube-api-access-25fmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:15:30.316360 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.316278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "213218d9-bbff-4fed-96d2-d9cf7ce2cd64" (UID: "213218d9-bbff-4fed-96d2-d9cf7ce2cd64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:15:30.415428 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.415398    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:30.415428 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.415429    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:30.415625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.415444    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:30.415625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.415452    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25fmn\" (UniqueName: \"kubernetes.io/projected/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-api-access-25fmn\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:30.415625 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.415460    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/213218d9-bbff-4fed-96d2-d9cf7ce2cd64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:30.417211 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.417187    2578 generic.go:356] "Generic (PLEG): container finished" podID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerID="77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4" exitCode=0
Sep 15 17:15:30.417278 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.417243    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"
Sep 15 17:15:30.417278 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.417247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerDied","Data":"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"}
Sep 15 17:15:30.417361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.417296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw" event={"ID":"213218d9-bbff-4fed-96d2-d9cf7ce2cd64","Type":"ContainerDied","Data":"abe4824707b73a316934a0f34be28fb3d3b99c0194e0c2d06355ccc7232606d3"}
Sep 15 17:15:30.417361 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.417324    2578 scope.go:117] "RemoveContainer" containerID="d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"
Sep 15 17:15:30.428947 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.427956    2578 scope.go:117] "RemoveContainer" containerID="77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"
Sep 15 17:15:30.437944 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.437926    2578 scope.go:117] "RemoveContainer" containerID="a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"
Sep 15 17:15:30.439538 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.439517    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:15:30.443192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.443173    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c9944f4dc-bdnjw"]
Sep 15 17:15:30.446055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446039    2578 scope.go:117] "RemoveContainer" containerID="d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"
Sep 15 17:15:30.446334 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:30.446316    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331\": container with ID starting with d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331 not found: ID does not exist" containerID="d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"
Sep 15 17:15:30.446386 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446346    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331"} err="failed to get container status \"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331\": rpc error: code = NotFound desc = could not find container \"d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331\": container with ID starting with d6467379c46292ad33e38b723a9c56dc3e92b79d63ff05234a5d401f6a5a5331 not found: ID does not exist"
Sep 15 17:15:30.446386 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446362    2578 scope.go:117] "RemoveContainer" containerID="77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"
Sep 15 17:15:30.446612 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:30.446591    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4\": container with ID starting with 77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4 not found: ID does not exist" containerID="77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"
Sep 15 17:15:30.446651 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446619    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4"} err="failed to get container status \"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4\": rpc error: code = NotFound desc = could not find container \"77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4\": container with ID starting with 77d18b233064ce625e1cad97aa7742bc8e10aa9721079cd9d366a8dfa25a3fa4 not found: ID does not exist"
Sep 15 17:15:30.446651 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446635    2578 scope.go:117] "RemoveContainer" containerID="a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"
Sep 15 17:15:30.446879 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:30.446862    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41\": container with ID starting with a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41 not found: ID does not exist" containerID="a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"
Sep 15 17:15:30.446948 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:30.446883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41"} err="failed to get container status \"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41\": rpc error: code = NotFound desc = could not find container \"a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41\": container with ID starting with a8d30dec14e19520b6418bc1bad890b1953979670e081d59004899681a3e6a41 not found: ID does not exist"
Sep 15 17:15:31.154224 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:31.154182    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" path="/var/lib/kubelet/pods/213218d9-bbff-4fed-96d2-d9cf7ce2cd64/volumes"
Sep 15 17:15:36.848648 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:36.848613    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:15:36.849133 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:36.848947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" containerID="cri-o://acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec" gracePeriod=30
Sep 15 17:15:36.849133 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:36.848992    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kube-rbac-proxy" containerID="cri-o://174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977" gracePeriod=30
Sep 15 17:15:37.452442 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.452413    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerID="174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977" exitCode=2
Sep 15 17:15:37.452614 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.452488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerDied","Data":"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"}
Sep 15 17:15:37.961794 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.961760    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962115    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kube-rbac-proxy"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962127    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kube-rbac-proxy"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="storage-initializer"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962171    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="storage-initializer"
Sep 15 17:15:37.962220 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962225    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kserve-container"
Sep 15 17:15:37.962448 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.962231    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="213218d9-bbff-4fed-96d2-d9cf7ce2cd64" containerName="kube-rbac-proxy"
Sep 15 17:15:37.965397 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.965371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:37.967874 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.967856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 15 17:15:37.968019 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.968002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:15:37.974479 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:37.974458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:15:38.085702 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.085669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.085939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.085708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.085939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.085773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.085939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.085836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2czf\" (UniqueName: \"kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.085939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.085885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.181152 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.181112    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 15 17:15:38.185452 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.185422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 15 17:15:38.186657 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.186629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2czf\" (UniqueName: \"kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.186809 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.186712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.186879 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.186834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.186879 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.186868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.186980 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.186906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.187835 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.187644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.187835 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.187721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.188051 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.188029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.189596 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.189576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.197484 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.197463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2czf\" (UniqueName: \"kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf\") pod \"isvc-pmml-runtime-predictor-885fc8777-ngcn8\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.275873 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.275844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:38.548808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:38.548780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:15:38.551444 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:15:38.551418    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fed42e_9825_49f8_a66c_4af3c0033e50.slice/crio-5594205628c6265b7d331e22c50419162741b2c190733fd200f40e4b299d65cd WatchSource:0}: Error finding container 5594205628c6265b7d331e22c50419162741b2c190733fd200f40e4b299d65cd: Status 404 returned error can't find the container with id 5594205628c6265b7d331e22c50419162741b2c190733fd200f40e4b299d65cd
Sep 15 17:15:39.463063 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.463020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerStarted","Data":"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"}
Sep 15 17:15:39.463482 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.463090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerStarted","Data":"5594205628c6265b7d331e22c50419162741b2c190733fd200f40e4b299d65cd"}
Sep 15 17:15:39.594754 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.594709    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:15:39.699987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.699905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") pod \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") "
Sep 15 17:15:39.699987 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.699964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert\") pod \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") "
Sep 15 17:15:39.700166 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.699993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pws2m\" (UniqueName: \"kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m\") pod \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") "
Sep 15 17:15:39.700166 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.700020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location\") pod \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") "
Sep 15 17:15:39.700166 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.700048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config\") pod \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\" (UID: \"3c421e01-c1dd-48ae-a2af-4827e1e5719e\") "
Sep 15 17:15:39.700445 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.700415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c421e01-c1dd-48ae-a2af-4827e1e5719e" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:15:39.700574 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.700412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c421e01-c1dd-48ae-a2af-4827e1e5719e" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:15:39.702234 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.702207    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c421e01-c1dd-48ae-a2af-4827e1e5719e" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:15:39.702327 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.702297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m" (OuterVolumeSpecName: "kube-api-access-pws2m") pod "3c421e01-c1dd-48ae-a2af-4827e1e5719e" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e"). InnerVolumeSpecName "kube-api-access-pws2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:15:39.709805 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.709710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c421e01-c1dd-48ae-a2af-4827e1e5719e" (UID: "3c421e01-c1dd-48ae-a2af-4827e1e5719e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:15:39.801563 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.801530    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:39.801563 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.801557    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pws2m\" (UniqueName: \"kubernetes.io/projected/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-api-access-pws2m\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:39.801563 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.801567    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:39.801808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.801575    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c421e01-c1dd-48ae-a2af-4827e1e5719e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:39.801808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:39.801585    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c421e01-c1dd-48ae-a2af-4827e1e5719e-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:15:40.469930 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.469895    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerID="acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec" exitCode=0
Sep 15 17:15:40.470336 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.469962    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"
Sep 15 17:15:40.470336 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.469972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerDied","Data":"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"}
Sep 15 17:15:40.470336 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.470034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk" event={"ID":"3c421e01-c1dd-48ae-a2af-4827e1e5719e","Type":"ContainerDied","Data":"01af64f9eb413a8c04fc980be7dc3465c8f904f3d5a055b6fcc3bcc01b94094b"}
Sep 15 17:15:40.470336 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.470060    2578 scope.go:117] "RemoveContainer" containerID="174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"
Sep 15 17:15:40.479251 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.479234    2578 scope.go:117] "RemoveContainer" containerID="acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"
Sep 15 17:15:40.486942 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.486925    2578 scope.go:117] "RemoveContainer" containerID="c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"
Sep 15 17:15:40.494822 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.494801    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:15:40.494992 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.494803    2578 scope.go:117] "RemoveContainer" containerID="174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"
Sep 15 17:15:40.495829 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:40.495728    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977\": container with ID starting with 174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977 not found: ID does not exist" containerID="174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"
Sep 15 17:15:40.495947 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.495860    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977"} err="failed to get container status \"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977\": rpc error: code = NotFound desc = could not find container \"174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977\": container with ID starting with 174adbabfd976465c0e7b308bfc1433abe528409340497a62def7027b15e7977 not found: ID does not exist"
Sep 15 17:15:40.495947 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.495889    2578 scope.go:117] "RemoveContainer" containerID="acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"
Sep 15 17:15:40.496279 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:40.496256    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec\": container with ID starting with acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec not found: ID does not exist" containerID="acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"
Sep 15 17:15:40.496346 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.496285    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec"} err="failed to get container status \"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec\": rpc error: code = NotFound desc = could not find container \"acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec\": container with ID starting with acc7f5f071e11487c2ecf259693aa35b69c18d1e68e4e28bb785e213d92041ec not found: ID does not exist"
Sep 15 17:15:40.496346 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.496302    2578 scope.go:117] "RemoveContainer" containerID="c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"
Sep 15 17:15:40.496585 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:15:40.496570    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e\": container with ID starting with c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e not found: ID does not exist" containerID="c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"
Sep 15 17:15:40.496628 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.496589    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e"} err="failed to get container status \"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e\": rpc error: code = NotFound desc = could not find container \"c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e\": container with ID starting with c670df8dec3cbf9ce3ec3b22486df4da2a2cbf45c34ca88ce3ffcb0638d8064e not found: ID does not exist"
Sep 15 17:15:40.498007 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:40.497990    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-q9ddk"]
Sep 15 17:15:41.154165 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:41.154131    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" path="/var/lib/kubelet/pods/3c421e01-c1dd-48ae-a2af-4827e1e5719e/volumes"
Sep 15 17:15:43.485160 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:43.485127    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerID="743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc" exitCode=0
Sep 15 17:15:43.485546 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:43.485168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerDied","Data":"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"}
Sep 15 17:15:51.533914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:51.533876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerStarted","Data":"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"}
Sep 15 17:15:51.534343 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:51.533941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerStarted","Data":"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"}
Sep 15 17:15:51.534343 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:51.534127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:51.553451 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:51.553393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podStartSLOduration=3.203521358 podStartE2EDuration="14.553372916s" totalImagesPullingTime="7.30203508s" totalInitContainerRuntime="4.047816478s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:15:51.551901245 +0000 UTC m=+2922.938540514" watchObservedRunningTime="2026-09-15 17:15:51.553372916 +0000 UTC m=+2922.940012183"
Sep 15 17:15:52.538664 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:52.538634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:52.539770 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:52.539721    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:15:53.542891 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:53.542849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:15:58.547657 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:58.547629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:15:58.548127 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:15:58.548100    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:08.549079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:08.549039    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:18.548124 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:18.548081    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:28.549069 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:28.549028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:38.548171 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:38.548136    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:48.548208 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:48.548168    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:16:58.548491 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:16:58.548442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 15 17:17:08.548890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:08.548851    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:17:19.165287 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:19.165244    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:17:19.165729 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:19.165516    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container" containerID="cri-o://9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3" gracePeriod=30
Sep 15 17:17:19.165841 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:19.165752    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kube-rbac-proxy" containerID="cri-o://6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5" gracePeriod=30
Sep 15 17:17:19.919929 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:19.919898    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerID="6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5" exitCode=2
Sep 15 17:17:19.920097 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:19.919968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerDied","Data":"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"}
Sep 15 17:17:22.822920 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.822895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:17:22.907356 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert\") pod \"e0fed42e-9825-49f8-a66c-4af3c0033e50\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") "
Sep 15 17:17:22.907502 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location\") pod \"e0fed42e-9825-49f8-a66c-4af3c0033e50\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") "
Sep 15 17:17:22.907502 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls\") pod \"e0fed42e-9825-49f8-a66c-4af3c0033e50\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") "
Sep 15 17:17:22.907502 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907426    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2czf\" (UniqueName: \"kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf\") pod \"e0fed42e-9825-49f8-a66c-4af3c0033e50\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") "
Sep 15 17:17:22.907502 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907462    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config\") pod \"e0fed42e-9825-49f8-a66c-4af3c0033e50\" (UID: \"e0fed42e-9825-49f8-a66c-4af3c0033e50\") "
Sep 15 17:17:22.907875 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0fed42e-9825-49f8-a66c-4af3c0033e50" (UID: "e0fed42e-9825-49f8-a66c-4af3c0033e50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:17:22.907975 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0fed42e-9825-49f8-a66c-4af3c0033e50" (UID: "e0fed42e-9825-49f8-a66c-4af3c0033e50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:17:22.907975 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.907965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0fed42e-9825-49f8-a66c-4af3c0033e50" (UID: "e0fed42e-9825-49f8-a66c-4af3c0033e50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:17:22.909700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.909664    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0fed42e-9825-49f8-a66c-4af3c0033e50" (UID: "e0fed42e-9825-49f8-a66c-4af3c0033e50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:17:22.910176 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.910134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf" (OuterVolumeSpecName: "kube-api-access-m2czf") pod "e0fed42e-9825-49f8-a66c-4af3c0033e50" (UID: "e0fed42e-9825-49f8-a66c-4af3c0033e50"). InnerVolumeSpecName "kube-api-access-m2czf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:17:22.934622 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.934595    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerID="9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3" exitCode=0
Sep 15 17:17:22.934776 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.934660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerDied","Data":"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"}
Sep 15 17:17:22.934776 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.934709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8" event={"ID":"e0fed42e-9825-49f8-a66c-4af3c0033e50","Type":"ContainerDied","Data":"5594205628c6265b7d331e22c50419162741b2c190733fd200f40e4b299d65cd"}
Sep 15 17:17:22.934776 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.934660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"
Sep 15 17:17:22.934776 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.934768    2578 scope.go:117] "RemoveContainer" containerID="6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"
Sep 15 17:17:22.943596 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.943579    2578 scope.go:117] "RemoveContainer" containerID="9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"
Sep 15 17:17:22.951484 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.951459    2578 scope.go:117] "RemoveContainer" containerID="743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"
Sep 15 17:17:22.956883 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.956860    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:17:22.961590 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.961565    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ngcn8"]
Sep 15 17:17:22.962995 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.962979    2578 scope.go:117] "RemoveContainer" containerID="6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"
Sep 15 17:17:22.963278 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:17:22.963261    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5\": container with ID starting with 6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5 not found: ID does not exist" containerID="6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"
Sep 15 17:17:22.963343 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.963284    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5"} err="failed to get container status \"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5\": rpc error: code = NotFound desc = could not find container \"6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5\": container with ID starting with 6ad97a9c8f24f0e5d6f14db32abe32aeca0d031a8893a2ec0937e84ce72c4cb5 not found: ID does not exist"
Sep 15 17:17:22.963343 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.963301    2578 scope.go:117] "RemoveContainer" containerID="9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"
Sep 15 17:17:22.963567 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:17:22.963545    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3\": container with ID starting with 9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3 not found: ID does not exist" containerID="9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"
Sep 15 17:17:22.963607 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.963574    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3"} err="failed to get container status \"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3\": rpc error: code = NotFound desc = could not find container \"9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3\": container with ID starting with 9d2d43d1664b52c8678d0023cfcbd8bcabf1c8f22e22b1664af76c00330b67c3 not found: ID does not exist"
Sep 15 17:17:22.963607 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.963591    2578 scope.go:117] "RemoveContainer" containerID="743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"
Sep 15 17:17:22.963874 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:17:22.963853    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc\": container with ID starting with 743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc not found: ID does not exist" containerID="743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"
Sep 15 17:17:22.963939 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:22.963879    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc"} err="failed to get container status \"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc\": rpc error: code = NotFound desc = could not find container \"743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc\": container with ID starting with 743ca663fc7743edd3a739b8db18354386f286bbf122be68b1f489df5b43fefc not found: ID does not exist"
Sep 15 17:17:23.009044 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.009022    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:17:23.009044 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.009040    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fed42e-9825-49f8-a66c-4af3c0033e50-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:17:23.009167 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.009049    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0fed42e-9825-49f8-a66c-4af3c0033e50-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:17:23.009167 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.009060    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2czf\" (UniqueName: \"kubernetes.io/projected/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-api-access-m2czf\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:17:23.009167 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.009068    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0fed42e-9825-49f8-a66c-4af3c0033e50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:17:23.154943 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:23.154909    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" path="/var/lib/kubelet/pods/e0fed42e-9825-49f8-a66c-4af3c0033e50/volumes"
Sep 15 17:17:24.293342 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293311    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293640    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293666    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293673    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="storage-initializer"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="storage-initializer"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293684    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kube-rbac-proxy"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293690    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kube-rbac-proxy"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293708    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="storage-initializer"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293715    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="storage-initializer"
Sep 15 17:17:24.293722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293725    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kube-rbac-proxy"
Sep 15 17:17:24.294203 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kube-rbac-proxy"
Sep 15 17:17:24.294203 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293830    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kube-rbac-proxy"
Sep 15 17:17:24.294203 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293840    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kserve-container"
Sep 15 17:17:24.294203 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293846    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0fed42e-9825-49f8-a66c-4af3c0033e50" containerName="kube-rbac-proxy"
Sep 15 17:17:24.294203 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.293851    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c421e01-c1dd-48ae-a2af-4827e1e5719e" containerName="kserve-container"
Sep 15 17:17:24.298109 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.298093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.300719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.300700    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:17:24.300719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.300717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:17:24.300924 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.300850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:17:24.300924 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.300869    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 15 17:17:24.300924 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.300860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:17:24.301524 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.301510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:17:24.301569 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.301554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 15 17:17:24.309371 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.309347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:17:24.420096 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.420053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.420289 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.420115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.420289 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.420180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwb7t\" (UniqueName: \"kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.420289 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.420239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.420289 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.420262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.521437 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.521437 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwb7t\" (UniqueName: \"kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.521681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.521681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.521681 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.522004 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.521976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.522154 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.522138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.522192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.522143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.524190 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.524172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.532015 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.531983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwb7t\" (UniqueName: \"kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.615949 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.615863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:24.888085 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.888056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:17:24.889918 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:17:24.889893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3974f3_8e1e_49a0_875b_e4f881a40e2e.slice/crio-4a9010baef5185eeddbe696f2c2429dedab8645f8b5d8ee6047853789c8d1714 WatchSource:0}: Error finding container 4a9010baef5185eeddbe696f2c2429dedab8645f8b5d8ee6047853789c8d1714: Status 404 returned error can't find the container with id 4a9010baef5185eeddbe696f2c2429dedab8645f8b5d8ee6047853789c8d1714
Sep 15 17:17:24.946169 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:24.946144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerStarted","Data":"4a9010baef5185eeddbe696f2c2429dedab8645f8b5d8ee6047853789c8d1714"}
Sep 15 17:17:25.951794 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:25.951731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerStarted","Data":"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"}
Sep 15 17:17:28.965432 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:28.965386    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerID="a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9" exitCode=0
Sep 15 17:17:28.965853 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:28.965458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerDied","Data":"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"}
Sep 15 17:17:29.971835 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:29.971786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerStarted","Data":"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"}
Sep 15 17:17:29.972221 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:29.971854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerStarted","Data":"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"}
Sep 15 17:17:29.972221 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:29.972103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:29.990919 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:29.990861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podStartSLOduration=2.106050977 podStartE2EDuration="5.990844138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.884793161s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:17:29.989458721 +0000 UTC m=+3021.376097991" watchObservedRunningTime="2026-09-15 17:17:29.990844138 +0000 UTC m=+3021.377483405"
Sep 15 17:17:30.977095 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:30.977064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:30.978406 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:30.978373    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:17:31.981775 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:31.981712    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:17:36.986021 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:36.985992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:17:36.986497 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:36.986473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:17:46.986751 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:46.986708    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:17:56.986613 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:17:56.986571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:06.986813 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:06.986692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:16.986461 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:16.986422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:26.986570 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:26.986530    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:36.987079 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:36.987037    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:46.987211 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:46.987177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:18:55.456353 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:55.456313    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:18:55.456783 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:55.456634    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" containerID="cri-o://5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156" gracePeriod=30
Sep 15 17:18:55.456783 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:55.456639    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kube-rbac-proxy" containerID="cri-o://0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0" gracePeriod=30
Sep 15 17:18:56.347569 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:56.347537    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerID="0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0" exitCode=2
Sep 15 17:18:56.347728 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:56.347580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerDied","Data":"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"}
Sep 15 17:18:56.982932 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:56.982887    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Sep 15 17:18:56.987283 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:56.987249    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 15 17:18:58.999137 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:58.999110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:18:59.033381 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwb7t\" (UniqueName: \"kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t\") pod \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") "
Sep 15 17:18:59.033540 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert\") pod \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") "
Sep 15 17:18:59.033540 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location\") pod \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") "
Sep 15 17:18:59.033631 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config\") pod \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") "
Sep 15 17:18:59.033687 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls\") pod \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\" (UID: \"0e3974f3-8e1e-49a0-875b-e4f881a40e2e\") "
Sep 15 17:18:59.033890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e3974f3-8e1e-49a0-875b-e4f881a40e2e" (UID: "0e3974f3-8e1e-49a0-875b-e4f881a40e2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:18:59.033890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e3974f3-8e1e-49a0-875b-e4f881a40e2e" (UID: "0e3974f3-8e1e-49a0-875b-e4f881a40e2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:18:59.034105 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.033924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e3974f3-8e1e-49a0-875b-e4f881a40e2e" (UID: "0e3974f3-8e1e-49a0-875b-e4f881a40e2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:18:59.034105 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.034008    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:18:59.034105 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.034025    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:18:59.034105 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.034035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:18:59.035724 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.035694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t" (OuterVolumeSpecName: "kube-api-access-mwb7t") pod "0e3974f3-8e1e-49a0-875b-e4f881a40e2e" (UID: "0e3974f3-8e1e-49a0-875b-e4f881a40e2e"). InnerVolumeSpecName "kube-api-access-mwb7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:18:59.035889 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.035786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e3974f3-8e1e-49a0-875b-e4f881a40e2e" (UID: "0e3974f3-8e1e-49a0-875b-e4f881a40e2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:18:59.134765 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.134725    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwb7t\" (UniqueName: \"kubernetes.io/projected/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-kube-api-access-mwb7t\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:18:59.134902 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.134768    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e3974f3-8e1e-49a0-875b-e4f881a40e2e-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:18:59.363158 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.363124    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerID="5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156" exitCode=0
Sep 15 17:18:59.363339 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.363183    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"
Sep 15 17:18:59.363339 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.363200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerDied","Data":"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"}
Sep 15 17:18:59.363339 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.363262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5" event={"ID":"0e3974f3-8e1e-49a0-875b-e4f881a40e2e","Type":"ContainerDied","Data":"4a9010baef5185eeddbe696f2c2429dedab8645f8b5d8ee6047853789c8d1714"}
Sep 15 17:18:59.363339 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.363291    2578 scope.go:117] "RemoveContainer" containerID="0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"
Sep 15 17:18:59.378130 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.378108    2578 scope.go:117] "RemoveContainer" containerID="5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"
Sep 15 17:18:59.382127 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.382093    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:18:59.386014 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.385955    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hp9t5"]
Sep 15 17:18:59.387029 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.387015    2578 scope.go:117] "RemoveContainer" containerID="a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"
Sep 15 17:18:59.394831 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.394815    2578 scope.go:117] "RemoveContainer" containerID="0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"
Sep 15 17:18:59.395110 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:18:59.395094    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0\": container with ID starting with 0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0 not found: ID does not exist" containerID="0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"
Sep 15 17:18:59.395157 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.395119    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0"} err="failed to get container status \"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0\": rpc error: code = NotFound desc = could not find container \"0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0\": container with ID starting with 0fe6971fefcc3e40a284b12b3528801d97005885de27735ea54b8dccd69f35e0 not found: ID does not exist"
Sep 15 17:18:59.395157 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.395136    2578 scope.go:117] "RemoveContainer" containerID="5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"
Sep 15 17:18:59.395407 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:18:59.395388    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156\": container with ID starting with 5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156 not found: ID does not exist" containerID="5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"
Sep 15 17:18:59.395462 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.395415    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156"} err="failed to get container status \"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156\": rpc error: code = NotFound desc = could not find container \"5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156\": container with ID starting with 5e74b6e922ab38b8fba792322333ccdbdb02cb8b8377f2f8036dc688eb6cf156 not found: ID does not exist"
Sep 15 17:18:59.395462 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.395435    2578 scope.go:117] "RemoveContainer" containerID="a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"
Sep 15 17:18:59.395639 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:18:59.395623    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9\": container with ID starting with a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9 not found: ID does not exist" containerID="a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"
Sep 15 17:18:59.395683 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:18:59.395642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9"} err="failed to get container status \"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9\": rpc error: code = NotFound desc = could not find container \"a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9\": container with ID starting with a2f0bab94a30e13315f23da6e0140d3f420688290d7ea1ddcd55d184ac5f46a9 not found: ID does not exist"
Sep 15 17:19:01.154093 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:19:01.154060    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" path="/var/lib/kubelet/pods/0e3974f3-8e1e-49a0-875b-e4f881a40e2e/volumes"
Sep 15 17:20:45.824438 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824401    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="storage-initializer"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824813    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="storage-initializer"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824838    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kube-rbac-proxy"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824845    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kube-rbac-proxy"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824860    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824866    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824939    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kserve-container"
Sep 15 17:20:45.825055 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.824947    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3974f3-8e1e-49a0-875b-e4f881a40e2e" containerName="kube-rbac-proxy"
Sep 15 17:20:45.828385 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.828365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:45.830942 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.830919    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cdd413\""
Sep 15 17:20:45.831080 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831047    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:20:45.831080 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831056    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:20:45.831208 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831047    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:20:45.831875 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831857    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cdd413-kube-rbac-proxy-sar-config\""
Sep 15 17:20:45.831982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cdd413-predictor-serving-cert\""
Sep 15 17:20:45.831982 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.831952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cdd413-dockercfg-gbvhj\""
Sep 15 17:20:45.838986 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.838963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:20:45.926025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.925982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:45.926192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.926037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:45.926192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.926076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:45.926192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.926107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhb4\" (UniqueName: \"kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:45.926192 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:45.926158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.026603 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.026561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.026603 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.026603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.026849 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.026627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhb4\" (UniqueName: \"kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.026849 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.026652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.026849 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.026699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.027032 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.027009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.027391 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.027340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.027391 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.027343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.029298 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.029274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.034565 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.034537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhb4\" (UniqueName: \"kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4\") pod \"isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") " pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.139251 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.139218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:20:46.411592 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.411569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:20:46.413564 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:20:46.413538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78204bf_6ce4_483e_bcd5_c8cf3db93931.slice/crio-c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b WatchSource:0}: Error finding container c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b: Status 404 returned error can't find the container with id c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b
Sep 15 17:20:46.837056 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.836962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerStarted","Data":"d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"}
Sep 15 17:20:46.837056 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:46.837030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerStarted","Data":"c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b"}
Sep 15 17:20:50.856619 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:50.856591    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/0.log"
Sep 15 17:20:50.857025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:50.856632    2578 generic.go:356] "Generic (PLEG): container finished" podID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerID="d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71" exitCode=1
Sep 15 17:20:50.857025 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:50.856714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerDied","Data":"d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"}
Sep 15 17:20:51.863382 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:51.863353    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/0.log"
Sep 15 17:20:51.863808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:51.863475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerStarted","Data":"a9cef6eac34851db3b7c291ae766e506fd2e8eb860001704ca722ac3c8930a34"}
Sep 15 17:20:56.891346 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.891317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/1.log"
Sep 15 17:20:56.891781 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.891764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/0.log"
Sep 15 17:20:56.891839 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.891800    2578 generic.go:356] "Generic (PLEG): container finished" podID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerID="a9cef6eac34851db3b7c291ae766e506fd2e8eb860001704ca722ac3c8930a34" exitCode=1
Sep 15 17:20:56.891878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.891839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerDied","Data":"a9cef6eac34851db3b7c291ae766e506fd2e8eb860001704ca722ac3c8930a34"}
Sep 15 17:20:56.891913 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.891893    2578 scope.go:117] "RemoveContainer" containerID="d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"
Sep 15 17:20:56.892252 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.892232    2578 scope.go:117] "RemoveContainer" containerID="d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"
Sep 15 17:20:56.902632 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:20:56.902601    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_kserve-ci-e2e-test(f78204bf-6ce4-483e-bcd5-c8cf3db93931)\"" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931"
Sep 15 17:20:56.902902 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:20:56.902876    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_kserve-ci-e2e-test_f78204bf-6ce4-483e-bcd5-c8cf3db93931_0 in pod sandbox c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b from index: no such id: 'd4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71'" containerID="d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"
Sep 15 17:20:56.902956 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:56.902910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_kserve-ci-e2e-test_f78204bf-6ce4-483e-bcd5-c8cf3db93931_0 in pod sandbox c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b from index: no such id: 'd4d4e66cd172ac28f8ceff778e4149fa7f585bc6c33827a2842703a3e6b14e71'"
Sep 15 17:20:57.897102 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:20:57.897072    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/1.log"
Sep 15 17:21:01.826834 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:01.826730    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:21:01.962174 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:01.962153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/1.log"
Sep 15 17:21:01.965460 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:01.965444    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:21:02.062519 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert\") pod \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") "
Sep 15 17:21:02.062685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvhb4\" (UniqueName: \"kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4\") pod \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") "
Sep 15 17:21:02.062685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls\") pod \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") "
Sep 15 17:21:02.062685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062676    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location\") pod \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") "
Sep 15 17:21:02.062848 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config\") pod \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\" (UID: \"f78204bf-6ce4-483e-bcd5-c8cf3db93931\") "
Sep 15 17:21:02.062961 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f78204bf-6ce4-483e-bcd5-c8cf3db93931" (UID: "f78204bf-6ce4-483e-bcd5-c8cf3db93931"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:21:02.063027 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.062978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f78204bf-6ce4-483e-bcd5-c8cf3db93931" (UID: "f78204bf-6ce4-483e-bcd5-c8cf3db93931"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:21:02.063167 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.063140    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f78204bf-6ce4-483e-bcd5-c8cf3db93931" (UID: "f78204bf-6ce4-483e-bcd5-c8cf3db93931"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:21:02.064869 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.064837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4" (OuterVolumeSpecName: "kube-api-access-fvhb4") pod "f78204bf-6ce4-483e-bcd5-c8cf3db93931" (UID: "f78204bf-6ce4-483e-bcd5-c8cf3db93931"). InnerVolumeSpecName "kube-api-access-fvhb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:21:02.064961 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.064910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f78204bf-6ce4-483e-bcd5-c8cf3db93931" (UID: "f78204bf-6ce4-483e-bcd5-c8cf3db93931"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:21:02.163641 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.163608    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-cabundle-cert\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:02.163641 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.163636    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhb4\" (UniqueName: \"kubernetes.io/projected/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-api-access-fvhb4\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:02.163860 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.163649    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f78204bf-6ce4-483e-bcd5-c8cf3db93931-proxy-tls\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:02.163860 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.163659    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kserve-provision-location\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:02.163860 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.163667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f78204bf-6ce4-483e-bcd5-c8cf3db93931-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:02.920704 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.920677    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs_f78204bf-6ce4-483e-bcd5-c8cf3db93931/storage-initializer/1.log"
Sep 15 17:21:02.921129 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.920806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"
Sep 15 17:21:02.921129 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.920812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs" event={"ID":"f78204bf-6ce4-483e-bcd5-c8cf3db93931","Type":"ContainerDied","Data":"c2308c254f5c34ae45c30f14f43a4b0095972313fda2011bd9c239240078032b"}
Sep 15 17:21:02.921129 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.920887    2578 scope.go:117] "RemoveContainer" containerID="a9cef6eac34851db3b7c291ae766e506fd2e8eb860001704ca722ac3c8930a34"
Sep 15 17:21:02.955679 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.955648    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:21:02.962407 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:02.962379    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cdd413-predictor-559bdd5ff5-7cpfs"]
Sep 15 17:21:03.154793 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:03.154763    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" path="/var/lib/kubelet/pods/f78204bf-6ce4-483e-bcd5-c8cf3db93931/volumes"
Sep 15 17:21:08.629109 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.629068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bwbzh/must-gather-7j5pq"]
Sep 15 17:21:08.629876 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.629851    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.629876 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.629878    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.630094 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.630079    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.630279 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.630264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.630348 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.630283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.630455 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.630441    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78204bf-6ce4-483e-bcd5-c8cf3db93931" containerName="storage-initializer"
Sep 15 17:21:08.635401 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.635377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.639084 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.638719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwbzh\"/\"openshift-service-ca.crt\""
Sep 15 17:21:08.639428 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.639362    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwbzh\"/\"kube-root-ca.crt\""
Sep 15 17:21:08.641177 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.641155    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwbzh/must-gather-7j5pq"]
Sep 15 17:21:08.717985 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.717943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c456b\" (UniqueName: \"kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.718147 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.717998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.819250 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.819211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c456b\" (UniqueName: \"kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.819410 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.819258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.819555 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.819541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.827046 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.827023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c456b\" (UniqueName: \"kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b\") pod \"must-gather-7j5pq\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") " pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:08.953306 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:08.953271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:09.222143 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:09.222064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwbzh/must-gather-7j5pq"]
Sep 15 17:21:09.225197 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:21:09.225173    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a83f34_fd7a_4a9e_b706_a2cf3b6ae381.slice/crio-e8015f8cff497a2e58022d80ef51ce65b893859b3338d408c4dc42a68d23ea6a WatchSource:0}: Error finding container e8015f8cff497a2e58022d80ef51ce65b893859b3338d408c4dc42a68d23ea6a: Status 404 returned error can't find the container with id e8015f8cff497a2e58022d80ef51ce65b893859b3338d408c4dc42a68d23ea6a
Sep 15 17:21:09.226773 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:09.226753    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 17:21:09.952141 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:09.952101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" event={"ID":"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381","Type":"ContainerStarted","Data":"e8015f8cff497a2e58022d80ef51ce65b893859b3338d408c4dc42a68d23ea6a"}
Sep 15 17:21:15.983792 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:15.983682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" event={"ID":"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381","Type":"ContainerStarted","Data":"7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7"}
Sep 15 17:21:15.983792 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:15.983757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" event={"ID":"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381","Type":"ContainerStarted","Data":"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"}
Sep 15 17:21:15.999674 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:15.999599    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" podStartSLOduration=1.7795048549999999 podStartE2EDuration="7.999578528s" totalImagesPullingTime="6.220073673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:21:15.998359807 +0000 UTC m=+3247.384999074" watchObservedRunningTime="2026-09-15 17:21:15.999578528 +0000 UTC m=+3247.386217796"
Sep 15 17:21:42.110656 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:42.110624    2578 generic.go:356] "Generic (PLEG): container finished" podID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerID="f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a" exitCode=0
Sep 15 17:21:42.111075 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:42.110701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" event={"ID":"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381","Type":"ContainerDied","Data":"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"}
Sep 15 17:21:42.111136 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:42.111088    2578 scope.go:117] "RemoveContainer" containerID="f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"
Sep 15 17:21:42.951700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:42.951657    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bwbzh_must-gather-7j5pq_e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381/gather/0.log"
Sep 15 17:21:43.748364 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.748330    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87zwh/must-gather-bjkc4"]
Sep 15 17:21:43.750986 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.750966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.753460 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.753430    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87zwh\"/\"openshift-service-ca.crt\""
Sep 15 17:21:43.753460 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.753445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87zwh\"/\"default-dockercfg-klvlx\""
Sep 15 17:21:43.753636 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.753445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87zwh\"/\"kube-root-ca.crt\""
Sep 15 17:21:43.760393 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.760373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/must-gather-bjkc4"]
Sep 15 17:21:43.841096 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.841052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8aaf60a0-9983-4cf6-a211-5c8deaf412de-must-gather-output\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.841096 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.841088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfxb\" (UniqueName: \"kubernetes.io/projected/8aaf60a0-9983-4cf6-a211-5c8deaf412de-kube-api-access-vcfxb\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.942531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.942484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8aaf60a0-9983-4cf6-a211-5c8deaf412de-must-gather-output\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.942700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.942537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfxb\" (UniqueName: \"kubernetes.io/projected/8aaf60a0-9983-4cf6-a211-5c8deaf412de-kube-api-access-vcfxb\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.942838 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.942820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8aaf60a0-9983-4cf6-a211-5c8deaf412de-must-gather-output\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:43.950379 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:43.950356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfxb\" (UniqueName: \"kubernetes.io/projected/8aaf60a0-9983-4cf6-a211-5c8deaf412de-kube-api-access-vcfxb\") pod \"must-gather-bjkc4\" (UID: \"8aaf60a0-9983-4cf6-a211-5c8deaf412de\") " pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:44.061513 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:44.061429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/must-gather-bjkc4"
Sep 15 17:21:44.333170 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:44.333148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/must-gather-bjkc4"]
Sep 15 17:21:44.335252 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:21:44.335224    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aaf60a0_9983_4cf6_a211_5c8deaf412de.slice/crio-4d0ee24b6e9fc897d8e566e9fac6f1fd6c33f93cf15cea9fac5b5daccc5cf3b0 WatchSource:0}: Error finding container 4d0ee24b6e9fc897d8e566e9fac6f1fd6c33f93cf15cea9fac5b5daccc5cf3b0: Status 404 returned error can't find the container with id 4d0ee24b6e9fc897d8e566e9fac6f1fd6c33f93cf15cea9fac5b5daccc5cf3b0
Sep 15 17:21:45.126375 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:45.126337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/must-gather-bjkc4" event={"ID":"8aaf60a0-9983-4cf6-a211-5c8deaf412de","Type":"ContainerStarted","Data":"4d0ee24b6e9fc897d8e566e9fac6f1fd6c33f93cf15cea9fac5b5daccc5cf3b0"}
Sep 15 17:21:46.134439 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:46.134393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/must-gather-bjkc4" event={"ID":"8aaf60a0-9983-4cf6-a211-5c8deaf412de","Type":"ContainerStarted","Data":"fe275323af42663ac1abff4e0419f3010d13f81efc0dbf0ec655558cd0458e2b"}
Sep 15 17:21:46.134955 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:46.134458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/must-gather-bjkc4" event={"ID":"8aaf60a0-9983-4cf6-a211-5c8deaf412de","Type":"ContainerStarted","Data":"b7066b963d81674292445d571ff77e36fa4ef9cef98195ede10f80974c08ec66"}
Sep 15 17:21:46.150065 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:46.150012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87zwh/must-gather-bjkc4" podStartSLOduration=2.316241783 podStartE2EDuration="3.149995566s" totalImagesPullingTime="833.753783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:21:46.149112802 +0000 UTC m=+3277.535752070" watchObservedRunningTime="2026-09-15 17:21:46.149995566 +0000 UTC m=+3277.536634833"
Sep 15 17:21:46.747792 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:46.747764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tpn5j_a16f1e9a-a3bb-450f-881a-15e066a3b639/global-pull-secret-syncer/0.log"
Sep 15 17:21:46.895368 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:46.895337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9snvm_2465ff10-35ea-4eb1-8440-c63a317977a7/konnectivity-agent/0.log"
Sep 15 17:21:47.004914 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:47.004844    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-27.ec2.internal_d09620bd92a67858915e7d86ee2bdcad/haproxy/0.log"
Sep 15 17:21:48.601434 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.601371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bwbzh/must-gather-7j5pq"]
Sep 15 17:21:48.601991 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.601644    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bwbzh/must-gather-7j5pq" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="copy" containerID="cri-o://7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7" gracePeriod=2
Sep 15 17:21:48.603895 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:48.603865    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7j5pq\" is forbidden: User \"system:node:ip-10-0-141-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bwbzh\": no relationship found between node 'ip-10-0-141-27.ec2.internal' and this object" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:48.605196 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.605172    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bwbzh/must-gather-7j5pq"]
Sep 15 17:21:48.953354 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.953060    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bwbzh_must-gather-7j5pq_e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381/copy/0.log"
Sep 15 17:21:48.956435 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.955862    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:48.963774 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:48.958946    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7j5pq\" is forbidden: User \"system:node:ip-10-0-141-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bwbzh\": no relationship found between node 'ip-10-0-141-27.ec2.internal' and this object" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:48.997247 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.997198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output\") pod \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") "
Sep 15 17:21:48.997441 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.997310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c456b\" (UniqueName: \"kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b\") pod \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\" (UID: \"e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381\") "
Sep 15 17:21:48.999679 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:48.999635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" (UID: "e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:21:49.008438 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.008389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b" (OuterVolumeSpecName: "kube-api-access-c456b") pod "e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" (UID: "e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381"). InnerVolumeSpecName "kube-api-access-c456b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:21:49.098245 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.098183    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-must-gather-output\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:49.098245 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.098219    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c456b\" (UniqueName: \"kubernetes.io/projected/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381-kube-api-access-c456b\") on node \"ip-10-0-141-27.ec2.internal\" DevicePath \"\""
Sep 15 17:21:49.154390 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.154361    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bwbzh_must-gather-7j5pq_e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381/copy/0.log"
Sep 15 17:21:49.155056 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.155032    2578 generic.go:356] "Generic (PLEG): container finished" podID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerID="7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7" exitCode=143
Sep 15 17:21:49.155255 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.155244    2578 scope.go:117] "RemoveContainer" containerID="7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7"
Sep 15 17:21:49.155503 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.155489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:49.166756 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:49.164025    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7j5pq\" is forbidden: User \"system:node:ip-10-0-141-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bwbzh\": no relationship found between node 'ip-10-0-141-27.ec2.internal' and this object" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:49.170327 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:49.170298    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-7j5pq\" is forbidden: User \"system:node:ip-10-0-141-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bwbzh\": no relationship found between node 'ip-10-0-141-27.ec2.internal' and this object" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" pod="openshift-must-gather-bwbzh/must-gather-7j5pq"
Sep 15 17:21:49.174044 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.174000    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" path="/var/lib/kubelet/pods/e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381/volumes"
Sep 15 17:21:49.184459 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.184372    2578 scope.go:117] "RemoveContainer" containerID="f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"
Sep 15 17:21:49.207419 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.207397    2578 scope.go:117] "RemoveContainer" containerID="7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7"
Sep 15 17:21:49.207853 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:49.207827    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7\": container with ID starting with 7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7 not found: ID does not exist" containerID="7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7"
Sep 15 17:21:49.207964 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.207866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7"} err="failed to get container status \"7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7\": rpc error: code = NotFound desc = could not find container \"7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7\": container with ID starting with 7c095e5ddab28f34e863ff7c5133e001e0a268f7b9ae4bcae3a85f08c827f7f7 not found: ID does not exist"
Sep 15 17:21:49.207964 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.207894    2578 scope.go:117] "RemoveContainer" containerID="f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"
Sep 15 17:21:49.208356 ip-10-0-141-27 kubenswrapper[2578]: E0915 17:21:49.208292    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a\": container with ID starting with f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a not found: ID does not exist" containerID="f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"
Sep 15 17:21:49.208356 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:49.208322    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a"} err="failed to get container status \"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a\": rpc error: code = NotFound desc = could not find container \"f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a\": container with ID starting with f0d93dd78f28831f1913017a10b936bb1f3b592d3dfdc7abdad9c7463639653a not found: ID does not exist"
Sep 15 17:21:50.907169 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:50.907080    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/alertmanager/0.log"
Sep 15 17:21:50.943878 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:50.943842    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/config-reloader/0.log"
Sep 15 17:21:50.987155 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:50.987127    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/kube-rbac-proxy-web/0.log"
Sep 15 17:21:51.020890 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.020865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/kube-rbac-proxy/0.log"
Sep 15 17:21:51.046588 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.046560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/kube-rbac-proxy-metric/0.log"
Sep 15 17:21:51.071405 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.071375    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/prom-label-proxy/0.log"
Sep 15 17:21:51.100674 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.100644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a46d88f4-70fd-4aef-854b-34667d405528/init-config-reloader/0.log"
Sep 15 17:21:51.184933 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.184841    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l5sw2_7c30c7ef-0a71-417a-b265-4265ac14fa19/kube-state-metrics/0.log"
Sep 15 17:21:51.207766 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.207698    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l5sw2_7c30c7ef-0a71-417a-b265-4265ac14fa19/kube-rbac-proxy-main/0.log"
Sep 15 17:21:51.232133 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.232103    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-l5sw2_7c30c7ef-0a71-417a-b265-4265ac14fa19/kube-rbac-proxy-self/0.log"
Sep 15 17:21:51.269291 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.269263    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c7d57d89c-b947v_d0ee721d-6d7f-4143-acb1-a4c7e239b884/metrics-server/0.log"
Sep 15 17:21:51.299029 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.299002    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-4fccx_10f31202-5e4e-4d41-8454-edb20c610fb8/monitoring-plugin/0.log"
Sep 15 17:21:51.336160 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.336123    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xs2s_ebd7b653-078c-45b6-9676-100d8f295184/node-exporter/0.log"
Sep 15 17:21:51.364003 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.363967    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xs2s_ebd7b653-078c-45b6-9676-100d8f295184/kube-rbac-proxy/0.log"
Sep 15 17:21:51.384685 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.384637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xs2s_ebd7b653-078c-45b6-9676-100d8f295184/init-textfile/0.log"
Sep 15 17:21:51.584925 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.584850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-psnts_5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a/kube-rbac-proxy-main/0.log"
Sep 15 17:21:51.612540 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.612482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-psnts_5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a/kube-rbac-proxy-self/0.log"
Sep 15 17:21:51.639454 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.639426    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-psnts_5ff148b5-87bc-4a6a-b4a2-8b50c9511d5a/openshift-state-metrics/0.log"
Sep 15 17:21:51.688696 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.688665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/prometheus/0.log"
Sep 15 17:21:51.715700 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.715668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/config-reloader/0.log"
Sep 15 17:21:51.740501 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.740471    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/thanos-sidecar/0.log"
Sep 15 17:21:51.765343 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.765282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/kube-rbac-proxy-web/0.log"
Sep 15 17:21:51.788019 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.787992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/kube-rbac-proxy/0.log"
Sep 15 17:21:51.811267 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.811239    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/kube-rbac-proxy-thanos/0.log"
Sep 15 17:21:51.836800 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.836694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5e20c9c2-fa7a-4d58-8061-33aedda36fb7/init-config-reloader/0.log"
Sep 15 17:21:51.867815 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.867756    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jqzh7_a84d5b9b-d058-4181-9a78-4218703965dc/prometheus-operator/0.log"
Sep 15 17:21:51.884808 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.884779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-jqzh7_a84d5b9b-d058-4181-9a78-4218703965dc/kube-rbac-proxy/0.log"
Sep 15 17:21:51.951149 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.951112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cff45487-wt6ks_8300868d-bede-4c69-89e4-e2d7abb76883/telemeter-client/0.log"
Sep 15 17:21:51.973595 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:51.973544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cff45487-wt6ks_8300868d-bede-4c69-89e4-e2d7abb76883/reload/0.log"
Sep 15 17:21:52.000532 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.000501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74cff45487-wt6ks_8300868d-bede-4c69-89e4-e2d7abb76883/kube-rbac-proxy/0.log"
Sep 15 17:21:52.034790 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.034760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/thanos-query/0.log"
Sep 15 17:21:52.055955 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.055929    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/kube-rbac-proxy-web/0.log"
Sep 15 17:21:52.080111 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.080086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/kube-rbac-proxy/0.log"
Sep 15 17:21:52.103015 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.102945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/prom-label-proxy/0.log"
Sep 15 17:21:52.128219 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.128186    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/kube-rbac-proxy-rules/0.log"
Sep 15 17:21:52.149507 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:52.149481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67559bd77c-77n4v_68b7a40b-3afc-4e5d-aa80-53d17c07500c/kube-rbac-proxy-metrics/0.log"
Sep 15 17:21:54.515719 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:54.515681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-x5vt9_9ca42511-83bd-42a7-a913-d80732f74f6e/download-server/0.log"
Sep 15 17:21:55.348555 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.348518    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"]
Sep 15 17:21:55.349091 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349070    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="gather"
Sep 15 17:21:55.349169 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349095    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="gather"
Sep 15 17:21:55.349169 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349112    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="copy"
Sep 15 17:21:55.349169 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349121    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="copy"
Sep 15 17:21:55.349297 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349235    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="copy"
Sep 15 17:21:55.349297 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.349248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a83f34-fd7a-4a9e-b706-a2cf3b6ae381" containerName="gather"
Sep 15 17:21:55.353071 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.353050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.362962 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.362935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"]
Sep 15 17:21:55.463516 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.463476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-proc\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.463722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.463533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-lib-modules\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.463722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.463570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-podres\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.463722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.463619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hds2\" (UniqueName: \"kubernetes.io/projected/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-kube-api-access-4hds2\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.463722 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.463647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-sys\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565139 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hds2\" (UniqueName: \"kubernetes.io/projected/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-kube-api-access-4hds2\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-sys\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-proc\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-lib-modules\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-podres\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565609 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-podres\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565908 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-sys\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.565986 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.565958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-proc\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.566083 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.566063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-lib-modules\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.574904 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.574881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hds2\" (UniqueName: \"kubernetes.io/projected/e4eaba7a-c615-4c55-93ec-f32bc9370cb6-kube-api-access-4hds2\") pod \"perf-node-gather-daemonset-vtbc6\" (UID: \"e4eaba7a-c615-4c55-93ec-f32bc9370cb6\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.667916 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.667876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:55.943157 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.943061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"]
Sep 15 17:21:55.948268 ip-10-0-141-27 kubenswrapper[2578]: W0915 17:21:55.948237    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode4eaba7a_c615_4c55_93ec_f32bc9370cb6.slice/crio-29cbdb8a9283acf79d2cf4520c5ab837c041c8e64fb6599b001554d5e6344182 WatchSource:0}: Error finding container 29cbdb8a9283acf79d2cf4520c5ab837c041c8e64fb6599b001554d5e6344182: Status 404 returned error can't find the container with id 29cbdb8a9283acf79d2cf4520c5ab837c041c8e64fb6599b001554d5e6344182
Sep 15 17:21:55.988835 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:55.988811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz7zt_1332b3c3-c602-48ba-9b74-42833d520538/dns/0.log"
Sep 15 17:21:56.007832 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.007796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz7zt_1332b3c3-c602-48ba-9b74-42833d520538/kube-rbac-proxy/0.log"
Sep 15 17:21:56.125067 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.125019    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kwhm5_34b7f025-99b9-4a22-a786-de64866c0071/dns-node-resolver/0.log"
Sep 15 17:21:56.204944 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.204852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6" event={"ID":"e4eaba7a-c615-4c55-93ec-f32bc9370cb6","Type":"ContainerStarted","Data":"8b974a135992844704ce25f5e3b47463b1f3061b12f684e3509bfe7ab42c1c38"}
Sep 15 17:21:56.204944 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.204926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6" event={"ID":"e4eaba7a-c615-4c55-93ec-f32bc9370cb6","Type":"ContainerStarted","Data":"29cbdb8a9283acf79d2cf4520c5ab837c041c8e64fb6599b001554d5e6344182"}
Sep 15 17:21:56.205176 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.204975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:21:56.220799 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.220705    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6" podStartSLOduration=1.220685339 podStartE2EDuration="1.220685339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 17:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:21:56.219581634 +0000 UTC m=+3287.606220922" watchObservedRunningTime="2026-09-15 17:21:56.220685339 +0000 UTC m=+3287.607324607"
Sep 15 17:21:56.640819 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:56.640796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-whwt5_932e1943-4d3d-42d7-bab6-7e4270936ec4/node-ca/0.log"
Sep 15 17:21:57.711635 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:57.711600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8s5gr_c72c1799-15bb-4c48-944b-ceee49f6af1c/serve-healthcheck-canary/0.log"
Sep 15 17:21:58.207475 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:58.207445    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6547_653f724c-fdde-40a4-8b73-876d52c2cc26/kube-rbac-proxy/0.log"
Sep 15 17:21:58.225366 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:58.225345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6547_653f724c-fdde-40a4-8b73-876d52c2cc26/exporter/0.log"
Sep 15 17:21:58.251686 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:21:58.251660    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6547_653f724c-fdde-40a4-8b73-876d52c2cc26/extractor/0.log"
Sep 15 17:22:00.976550 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:00.976516    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-vmf5j_a5983d46-8f2f-4cb6-bebe-3c3433a60148/manager/0.log"
Sep 15 17:22:01.043435 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.043403    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g4db8_c1db47f0-f55c-4445-a648-32d814b3f1e3/s3-init/0.log"
Sep 15 17:22:01.058977 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.058953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g4db8_c1db47f0-f55c-4445-a648-32d814b3f1e3/download-hf-model/0.log"
Sep 15 17:22:01.088149 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.088122    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-szxbz_f072f822-1ca4-4c34-bef3-c7d6e529b45f/s3-tls-init-custom/0.log"
Sep 15 17:22:01.102726 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.102701    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-szxbz_f072f822-1ca4-4c34-bef3-c7d6e529b45f/download-hf-model/0.log"
Sep 15 17:22:01.129225 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.129202    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-szv5s_7dc33fd4-fb02-4858-86ec-d2dbcd386344/s3-tls-init-serving/0.log"
Sep 15 17:22:01.143439 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.143418    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-szv5s_7dc33fd4-fb02-4858-86ec-d2dbcd386344/download-hf-model/0.log"
Sep 15 17:22:01.214106 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:01.214076    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-5h76h_12f087e3-0c3d-4d6a-ae96-dce5cc97cbc6/seaweedfs-tls-serving/0.log"
Sep 15 17:22:02.222154 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:02.222128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-vtbc6"
Sep 15 17:22:06.475089 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.475051    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/kube-multus-additional-cni-plugins/0.log"
Sep 15 17:22:06.494341 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.494315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/egress-router-binary-copy/0.log"
Sep 15 17:22:06.516853 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.516829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/cni-plugins/0.log"
Sep 15 17:22:06.539689 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.539663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/bond-cni-plugin/0.log"
Sep 15 17:22:06.559603 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.559572    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/routeoverride-cni/0.log"
Sep 15 17:22:06.580175 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.580121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/whereabouts-cni-bincopy/0.log"
Sep 15 17:22:06.603490 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:06.603460    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4nsv5_94305cb9-6f3f-4231-9211-c4bac29600f7/whereabouts-cni/0.log"
Sep 15 17:22:07.012661 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:07.012631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xsxqb_24ee9eeb-a6e5-4207-b5a1-a84b427c76bd/kube-multus/0.log"
Sep 15 17:22:07.122531 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:07.122495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjswr_b15f4535-2258-4631-9616-f79019a4f309/network-metrics-daemon/0.log"
Sep 15 17:22:07.141481 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:07.141454    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjswr_b15f4535-2258-4631-9616-f79019a4f309/kube-rbac-proxy/0.log"
Sep 15 17:22:08.082432 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.082400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/ovn-controller/0.log"
Sep 15 17:22:08.098409 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.098385    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/ovn-acl-logging/0.log"
Sep 15 17:22:08.118311 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.118287    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/kube-rbac-proxy-node/0.log"
Sep 15 17:22:08.136968 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.136943    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 17:22:08.155754 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.155713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/northd/0.log"
Sep 15 17:22:08.174898 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.174874    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/nbdb/0.log"
Sep 15 17:22:08.196629 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.196597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/sbdb/0.log"
Sep 15 17:22:08.249430 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:08.249402    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtgkt_a3aa3f2c-74b4-43cd-9c9a-d08ae3a1e590/ovnkube-controller/0.log"
Sep 15 17:22:09.502117 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:09.502080    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q5g8m_656cb863-0d56-4cec-838c-6f8a0bbdba5e/network-check-target-container/0.log"
Sep 15 17:22:10.399503 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:10.399479    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-d52kx_54a962b6-0a66-47b4-8205-4bd659c5c8e0/iptables-alerter/0.log"
Sep 15 17:22:11.034404 ip-10-0-141-27 kubenswrapper[2578]: I0915 17:22:11.034368    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-g28k2_fa5880e1-7205-4d98-b561-c00b65624313/tuned/0.log"