Sep 16 18:40:15.089587 ip-10-0-136-81 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:15.489717 ip-10-0-136-81 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:15.489717 ip-10-0-136-81 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:15.489717 ip-10-0-136-81 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:15.489717 ip-10-0-136-81 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:15.489717 ip-10-0-136-81 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:15.490429 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.489732    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:15.493165 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493147    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.493165 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493165    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493169    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493172    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493175    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493178    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493187    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493189    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493192    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493195    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493197    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493200    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493203    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493205    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493208    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493211    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493213    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493215    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493218    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493220    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493223    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.493231 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493226    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493228    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493232    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493241    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493246    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493250    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493253    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493256    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493258    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493261    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493263    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493266    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493268    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493271    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493273    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493276    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493281    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493285    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493289    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.493733 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493292    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493296    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493300    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493316    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493319    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493322    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493324    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493327    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493331    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493334    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493336    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493339    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493341    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493344    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493346    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493350    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493353    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493355    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493358    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.494197 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493361    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493365    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493368    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493372    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493374    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493377    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493380    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493383    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493386    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493389    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493391    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493394    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493397    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493400    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493403    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493405    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493407    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493410    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493412    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.494683 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493415    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493417    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493420    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493423    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493425    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493427    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493430    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493433    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493435    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493438    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493441    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493443    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493446    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493448    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493451    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493454    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493456    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493459    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493461    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493464    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.495174 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.493466    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494167    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494175    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494180    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494183    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494186    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494189    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494193    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494196    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494199    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494202    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494205    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494207    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494211    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494213    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494216    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494220    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494223    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494227    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.495695 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494230    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494233    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494236    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494239    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494241    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494244    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494247    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494249    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494252    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494254    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494257    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494259    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494262    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494264    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494267    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494270    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494272    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494275    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494277    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494280    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.496177 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494282    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494285    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494288    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494291    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494295    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494297    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494300    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494321    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494325    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494329    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494333    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494338    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494342    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494346    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494350    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494354    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494358    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494361    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494364    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.496788 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494367    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494370    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494372    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494375    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494378    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494380    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494382    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494385    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494388    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494390    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494393    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494395    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494398    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494400    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494402    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494406    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494408    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494411    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494413    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494417    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.497284 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494420    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494422    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494425    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494427    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494430    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494433    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494435    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494438    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494441    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494443    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494446    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494448    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494451    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494453    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494456    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494459    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494461    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494463    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494466    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494469    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.497791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494471    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494473    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494832    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494836    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494838    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494841    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494844    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494847    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494850    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494852    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494855    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494859    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494862    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494864    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494867    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494869    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494872    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494875    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494877    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494880    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.498277 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494883    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494885    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494888    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494890    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494893    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494895    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494898    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494901    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494903    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494906    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494908    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494910    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494913    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494916    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494919    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494922    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494926    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494929    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494932    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.498791 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494934    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494937    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494939    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494942    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494945    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494947    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494950    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494953    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494955    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494958    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494961    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494963    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494966    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494968    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494971    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494973    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494976    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494978    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494981    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494983    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.499260 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494985    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494990    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494993    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494996    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.494999    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495001    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495004    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495006    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495009    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495011    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495014    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495016    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495018    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495021    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495024    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495026    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495029    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495032    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495034    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.499754 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495037    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495039    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495042    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495045    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495049    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495053    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495056    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495059    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495061    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495064    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495067    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495069    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495072    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495075    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495077    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495080    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495082    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495085    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495087    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.500285 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495089    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495092    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495094    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495097    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495173    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495211    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495217    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495222    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495226    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495230    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495236    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495240    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495243    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495247    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495250    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495255    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495258    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495261    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495264    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495267    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495270    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495273    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495277    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:15.500765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495280    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495283    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495286    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495289    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495293    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495296    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495299    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495319    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495323    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495328    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495332    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495335    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495338    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495342    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495345    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495348    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495351    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495354    2579 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495357    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495363    2579 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495366    2579 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495369    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495372    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495375    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:15.501340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495380    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495383    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495386    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495389    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495392    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495395    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495398    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495401    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495404    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495407    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495410    2579 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495414    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495417    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495419    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495422    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495425    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495428    2579 flags.go:64] FLAG: --help="false"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495431    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495434    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495437    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495440    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495444    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495447    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495450    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:15.501930 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495453    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495456    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495459    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495463    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495466    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495469    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495472    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495475    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495480    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495483    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495487    2579 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495489    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495492    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495495    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495500    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495503    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495506    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495509    2579 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495511    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495515    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495518    2579 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495520    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495525    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495527    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495531    2579 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:15.502594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495534    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495537    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495540    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495543    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495546    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495549    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495552    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495559    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495562    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495565    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495569    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495572    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495578    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495581    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495584    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495587    2579 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495591    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495594    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-022b3afccdc6c2032"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495597    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495599    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495603    2579 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495606    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495609    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495612    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:15.503204 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495615    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495618    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495622    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495625    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495628    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495631    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495634    2579 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495637    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495640    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495643    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495646    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495648    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495651    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495654    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495657    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495660    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495663    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495666    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495670    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495673    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495676    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495679    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495684    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495687    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495692    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:15.504064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495698    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495701    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495704    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495707    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495710    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495713    2579 flags.go:64] FLAG: --v="2"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495717    2579 flags.go:64] FLAG: --version="false"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495721    2579 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495725    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.495728    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495844    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495848    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495851    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495854    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495857    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495859    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495862    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495864    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495867    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495869    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495872    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495874    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.504982 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495876    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495879    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495881    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495885    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495888    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495890    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495892    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495896    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495898    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495902    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495905    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495907    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495910    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495913    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495916    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495918    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495921    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495924    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495926    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495929    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.505554 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495933    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495936    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495939    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495943    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495946    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495949    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495952    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495955    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495957    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495960    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495962    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495965    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495967    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495970    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495973    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495976    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495978    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495981    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495983    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.506044 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495986    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495989    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495993    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495996    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.495999    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496001    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496004    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496006    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496009    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496011    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496014    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496017    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496019    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496022    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496024    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496027    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496029    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496032    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496034    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.506546 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496037    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496040    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496042    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496045    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496047    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496050    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496052    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496055    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496057    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496061    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496064    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496067    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496069    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496072    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496074    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496078    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496081    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496083    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496086    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496088    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.507022 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496091    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496093    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496096    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496098    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496101    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496103    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496106    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496108    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.496111    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.496661    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.505413    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.505563    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.506359    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.506370    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506464    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.507560 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506469    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506473    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506476    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506480    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506483    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506486    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506488    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506491    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506494    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506497    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506500    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506502    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506505    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506508    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506511    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506513    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506516    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506518    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506521    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506523    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.507946 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506526    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506529    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506531    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506534    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506537    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506540    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506542    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506549    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506553    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506557    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506559    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506562    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506565    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506567    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506570    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506573    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506575    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506578    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506580    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506583    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.508452 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506585    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506588    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506591    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506593    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506596    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506599    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506602    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506604    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506607    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506609    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506612    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506614    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506617    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506620    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506622    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506626    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506630    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506634    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506637    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506639    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.508949 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506643    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506645    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506648    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506650    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506653    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506656    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506658    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506661    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506663    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506666    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506669    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506671    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506674    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506677    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506681    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506684    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506688    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506691    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506694    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.509467 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506697    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506700    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506704    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506707    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506709    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506712    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506715    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506717    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506720    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506722    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506724    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506728    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506730    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506733    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506736    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506738    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506741    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506743    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.509932 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506746    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.506751    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506871    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506877    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506879    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506882    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506885    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506888    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506890    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506893    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506900    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506903    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506906    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506909    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506912    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:15.510388 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506915    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506918    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506921    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506923    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506926    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506929    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506932    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506934    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506937    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506939    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506942    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506944    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506947    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506950    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506953    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506955    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506958    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506961    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506963    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506966    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:15.510826 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506968    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506971    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506975    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506979    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506981    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506984    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506987    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506990    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506992    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506995    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.506998    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507001    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507004    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507007    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507010    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507013    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507016    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507020    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507023    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:15.511432 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507027    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507031    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507034    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507037    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507040    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507043    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507046    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507049    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507051    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507054    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507057    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507059    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507062    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507064    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507067    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507069    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507072    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507074    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507077    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507079    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:15.511924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507082    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507084    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507087    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507089    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507092    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507094    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507100    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507103    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507106    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507108    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507111    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507113    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507116    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507118    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507121    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507126    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507128    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507131    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507133    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507136    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:15.512534 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507138    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507141    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507144    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507146    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507149    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507151    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:15.507154    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.507159    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.507275    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.510230    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.511164    2579 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.511278    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:15.513226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.511337    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:15.513827 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.513630    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:15.514844 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.514824    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:15.522739 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.522720    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:15.526176 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.526149    2579 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:15.531018 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.530908    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:15.533139 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.533112    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81ad81f6-515d-4105-adf5-b6aa5e9612af:/dev/nvme0n1p3 a39e38b0-eb29-4432-a3b0-f567a1acd96d:/dev/nvme0n1p4]
Sep 16 18:40:15.533216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.533137    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:15.540942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.540840    2579 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:15.540371203 +0000 UTC m=+0.333617101 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103841 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa48745249b12fcef251495b79cb2 SystemUUID:ec2fa487-4524-9b12-fcef-251495b79cb2 BootID:3b23357b-1096-4841-ab7a-8ede076289dd Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:2b:67:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:2b:67:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:34:51:72:5a:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:15.540942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.540936    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:15.541062 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.541047    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:15.543036 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.543009    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:15.543194 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.543038    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:15.544378 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.544364    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:15.544686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.544646    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:15.544686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.544680    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:15.544790 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.544704    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:15.545987 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.545918    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:15.547055 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.547035    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:15.549099 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549085    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:15.549403 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549392    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:15.549438 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549426    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:15.549478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549439    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:15.549478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549457    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:15.549478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549470    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:15.549552 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549491    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:15.549845 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549833    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:15.549889 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.549851    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:15.552920 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.552904    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:15.555035 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.555022    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:15.555084 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.555052    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:15.557684 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557672    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:15.557729 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557694    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:15.557729 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557702    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:15.557729 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557707    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:15.557729 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557716    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:15.557729 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557721    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557734    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557740    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557746    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557751    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557759    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:15.557854 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.557765    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:15.558487 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.558478    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:15.558487 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.558488    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:15.561756 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.561739    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:15.562209 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.562188    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:15.562242 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.562195    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:15.562459 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.562447    2579 server.go:1268] "Started kubelet"
Sep 16 18:40:15.562628 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.562599    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:15.562718 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.562647    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:15.563231 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.563187    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:15.563359 ip-10-0-136-81 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:15.564199 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.564042    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:15.566143 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.565281    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:15.569093 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.569066    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:15.570366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.570353    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:15.570366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.570360    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:15.570616 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.570587    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.570999 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.570980    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:15.570999 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.570981    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:15.571131 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.571010    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:15.571131 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.571071    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:15.571131 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.571080    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:15.571838 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.571820    2579 factory.go:55] Registering systemd factory
Sep 16 18:40:15.571838 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.571837    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:15.572014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572004    2579 factory.go:153] Registering CRI-O factory
Sep 16 18:40:15.572014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572015    2579 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:15.572089 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572057    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:15.572089 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572072    2579 factory.go:103] Registering Raw factory
Sep 16 18:40:15.572089 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572081    2579 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:15.572424 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.572402    2579 manager.go:349] Starting recovery of all containers
Sep 16 18:40:15.572517 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.572479    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:15.572629 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.572608    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:15.574107 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.572474    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-81.ec2.internal.18d5e12e3750a515  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-81.ec2.internal,UID:ip-10-0-136-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-81.ec2.internal,},FirstTimestamp:2026-09-16 18:40:15.562417429 +0000 UTC m=+0.355663326,LastTimestamp:2026-09-16 18:40:15.562417429 +0000 UTC m=+0.355663326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-81.ec2.internal,}"
Sep 16 18:40:15.586540 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.586517    2579 manager.go:354] Recovery completed
Sep 16 18:40:15.586633 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.586596    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-26wvf"
Sep 16 18:40:15.590951 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.590937    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.592800 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.592785    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.592853 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.592815    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.592853 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.592825    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.593031 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.593012    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-26wvf"
Sep 16 18:40:15.593497 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.593486    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:15.593539 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.593499    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:15.593539 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.593516    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:15.593539 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.593515    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.594952 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.594935    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.595098 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.594967    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.595098 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.594979    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.595183 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.595142    2579 policy_none.go:49] "None policy: Start"
Sep 16 18:40:15.595183 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.595160    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:15.595183 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.595175    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:15.595719 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.595230    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-81.ec2.internal.18d5e12e39204783  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-81.ec2.internal,UID:ip-10-0-136-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-81.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-81.ec2.internal,},FirstTimestamp:2026-09-16 18:40:15.592802179 +0000 UTC m=+0.386048077,LastTimestamp:2026-09-16 18:40:15.592802179 +0000 UTC m=+0.386048077,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-81.ec2.internal,}"
Sep 16 18:40:15.596544 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.596531    2579 policy_none.go:47] "Start"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632434    2579 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.632466    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632475    2579 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632717    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632727    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632826    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632903    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.632910    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.633498    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:15.642441 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.633541    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.667472 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.667440    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:15.668872 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.668847    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:15.668973 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.668897    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:15.668973 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.668923    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:15.668973 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.668961    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:15.671026 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.670990    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:15.733124 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.733095    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.735343 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.735325    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.735435 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.735392    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.735435 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.735409    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.735532 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.735438    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.744582 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.744506    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.744582 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.744529    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-81.ec2.internal\": node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.760756 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.760731    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.769822 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.769790    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"]
Sep 16 18:40:15.769893 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.769874    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.770746 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.770731    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.770807 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.770765    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.770807 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.770776    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.772096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772084    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.772217 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.772256 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772231    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.772851 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772836    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.772929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772854    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.772929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772865    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.772929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772877    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.772929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772892    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.773091 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.772878    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.774205 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.774190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.774268 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.774217    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:15.775098 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.775084    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:15.775194 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.775107    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:15.775194 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.775122    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:15.802055 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.802030    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-81.ec2.internal\" not found" node="ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.806704 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.806690    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-81.ec2.internal\" not found" node="ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.861022 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.860988    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.872821 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.872794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.872923 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.872822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.872923 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.872842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d64174e840dde57cd70ce29f4e8fd34-config\") pod \"kube-apiserver-proxy-ip-10-0-136-81.ec2.internal\" (UID: \"2d64174e840dde57cd70ce29f4e8fd34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.961135 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:15.961102    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:15.973531 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d64174e840dde57cd70ce29f4e8fd34-config\") pod \"kube-apiserver-proxy-ip-10-0-136-81.ec2.internal\" (UID: \"2d64174e840dde57cd70ce29f4e8fd34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.973639 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.973639 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.973748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.973748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d64174e840dde57cd70ce29f4e8fd34-config\") pod \"kube-apiserver-proxy-ip-10-0-136-81.ec2.internal\" (UID: \"2d64174e840dde57cd70ce29f4e8fd34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:15.973748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:15.973737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d621e84069d8ebb64c644982a8593730-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal\" (UID: \"d621e84069d8ebb64c644982a8593730\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:16.061678 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:16.061576    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:16.105100 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.105064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:16.110053 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.110038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:16.162546 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:16.162517    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:16.263125 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:16.263087    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:16.363637 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:16.363561    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:16.443336 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.443293    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:16.464093 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:16.464066    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-81.ec2.internal\" not found"
Sep 16 18:40:16.511570 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.511543    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:16.512014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.511723    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:16.512014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.511729    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:16.551290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.551271    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:16.571326 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.571286    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:16.571430 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.571290    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:16.582120 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.582102    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:16.582965 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.582951    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal"
Sep 16 18:40:16.584425 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.584408    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:16.592383 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.592366    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:16.596444 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.596420    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:16 +0000 UTC" deadline="2028-04-13 05:46:47.201580509 +0000 UTC"
Sep 16 18:40:16.596444 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.596444    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13787h6m30.605138974s"
Sep 16 18:40:16.607913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.607892    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9z8bh"
Sep 16 18:40:16.614927 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.614882    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9z8bh"
Sep 16 18:40:16.685817 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:16.685649    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d64174e840dde57cd70ce29f4e8fd34.slice/crio-fdb0ca16aebb15083094d0818fb19c9768a3191ecf8e28caa69b4d4359229832 WatchSource:0}: Error finding container fdb0ca16aebb15083094d0818fb19c9768a3191ecf8e28caa69b4d4359229832: Status 404 returned error can't find the container with id fdb0ca16aebb15083094d0818fb19c9768a3191ecf8e28caa69b4d4359229832
Sep 16 18:40:16.690465 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.690452    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:16.713520 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:16.713496    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd621e84069d8ebb64c644982a8593730.slice/crio-fbc7b6dad86ae6c94e9af1e67658f7ff74053d64d9fdea63cd9bac78172af1e7 WatchSource:0}: Error finding container fbc7b6dad86ae6c94e9af1e67658f7ff74053d64d9fdea63cd9bac78172af1e7: Status 404 returned error can't find the container with id fbc7b6dad86ae6c94e9af1e67658f7ff74053d64d9fdea63cd9bac78172af1e7
Sep 16 18:40:16.807080 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:16.807054    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:18.624540 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.624508    2579 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:18.632570 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.632544    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:18.635026 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.635002    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal","openshift-multus/network-metrics-daemon-bc7ds","kube-system/global-pull-secret-syncer-h69vp","openshift-multus/multus-4chnh","openshift-multus/multus-additional-cni-plugins-xmplp","openshift-network-diagnostics/network-check-target-cdlgj","openshift-network-operator/iptables-alerter-6p5cm","openshift-ovn-kubernetes/ovnkube-node-tn8vm","kube-system/konnectivity-agent-d65wk","kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h","openshift-cluster-node-tuning-operator/tuned-ktv6k","openshift-dns/node-resolver-mh7lg","openshift-image-registry/node-ca-fjkkt"]
Sep 16 18:40:18.636396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.636376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.637528 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.637505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.638788 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.638766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.638880 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.638855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:18.638989 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.638949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:18.638989 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.638948    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.639267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.639250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.639267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.639261    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wvflj\""
Sep 16 18:40:18.640067 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.640046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.640067 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.640053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ngzcs\""
Sep 16 18:40:18.640207 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.640082    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.640288 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.640267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.641306 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.641290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.642443 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.642426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:18.642558 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.642521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:18.644319 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.643560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.645765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.645750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.646834 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.646819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.648448 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.648401    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vjbpc\""
Sep 16 18:40:18.648448 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.648428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:18.649166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.648428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:18.649166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.648674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.649166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:18.649166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zb5df\""
Sep 16 18:40:18.649640 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.649724 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649626    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:18.649923 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649901    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:18.649989 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.649904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:18.650430 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.650393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:18.650613 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.650593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:18.650705 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.650629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-txhpc\""
Sep 16 18:40:18.650761 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.650741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.651199 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.651029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rczbp\""
Sep 16 18:40:18.651199 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.651163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.651879 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.651861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.653508 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.653483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.654433 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:18.654433 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.654626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:18.654626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:18.654789 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7xrx\""
Sep 16 18:40:18.654901 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.654869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.655689 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.655667    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:18.655809 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.655794    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.656043 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.656163 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656047    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:18.656163 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87z7n\""
Sep 16 18:40:18.656290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.656290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9f51084-955f-4058-b1e2-2da5bcea710b-agent-certs\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.656290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/786b0e59-4b3f-47c7-9c97-cda697025943-host\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc49d\" (UniqueName: \"kubernetes.io/projected/786b0e59-4b3f-47c7-9c97-cda697025943-kube-api-access-sc49d\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b44a16f-1b5d-46db-a58f-1ac32c313498-iptables-alerter-script\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8567d\" (UniqueName: \"kubernetes.io/projected/eb3c189b-2662-4686-accc-a20201cd7ef3-kube-api-access-8567d\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.656447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-kubelet-config\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.656694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-cni-binary-copy\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwsb\" (UniqueName: \"kubernetes.io/projected/5b44a16f-1b5d-46db-a58f-1ac32c313498-kube-api-access-cvwsb\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.656694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-tmp-dir\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.656694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-multus\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-hostroot\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-system-cni-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-hosts-file\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-dbus\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-k8s-cni-cncf-io\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-bin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-etc-kubernetes\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.656926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9f51084-955f-4058-b1e2-2da5bcea710b-konnectivity-ca\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.656959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b44a16f-1b5d-46db-a58f-1ac32c313498-host-slash\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-netns\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-daemon-config\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8w4q\" (UniqueName: \"kubernetes.io/projected/932754fd-04f5-4fc3-bd81-36b598b7311a-kube-api-access-k8w4q\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-system-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-kubelet\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-multus-certs\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-cnibin\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/786b0e59-4b3f-47c7-9c97-cda697025943-serviceca\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7vm\" (UniqueName: \"kubernetes.io/projected/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-kube-api-access-gz7vm\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.657420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-conf-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.658112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-cnibin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.658112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:18.658112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-socket-dir-parent\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.658410 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.657849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cd458\""
Sep 16 18:40:18.691048 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.691017    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:17 +0000 UTC" deadline="2028-02-29 09:08:09.920323062 +0000 UTC"
Sep 16 18:40:18.691048 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.691049    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12734h27m51.229278931s"
Sep 16 18:40:18.746206 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.746177    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:18.746517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.746489    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:18.748218 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.748172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal" event={"ID":"d621e84069d8ebb64c644982a8593730","Type":"ContainerStarted","Data":"fbc7b6dad86ae6c94e9af1e67658f7ff74053d64d9fdea63cd9bac78172af1e7"}
Sep 16 18:40:18.749408 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.749383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal" event={"ID":"2d64174e840dde57cd70ce29f4e8fd34","Type":"ContainerStarted","Data":"fdb0ca16aebb15083094d0818fb19c9768a3191ecf8e28caa69b4d4359229832"}
Sep 16 18:40:18.758724 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.758839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.758839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-tmp-dir\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.758839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-multus\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.758839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-etc-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.758839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-lib-modules\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-dbus\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-kubernetes\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-run\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9f51084-955f-4058-b1e2-2da5bcea710b-konnectivity-ca\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-multus\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8w4q\" (UniqueName: \"kubernetes.io/projected/932754fd-04f5-4fc3-bd81-36b598b7311a-kube-api-access-k8w4q\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-netns\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.758999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-script-lib\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-systemd\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-sys\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.759095 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-dbus\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-system-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-tmp-dir\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-ovn\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-node-log\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-system-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-config\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-device-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-cnibin\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-conf-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysconfig\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c9f51084-955f-4058-b1e2-2da5bcea710b-konnectivity-ca\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-var-lib-kubelet\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.759806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.759568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9f51084-955f-4058-b1e2-2da5bcea710b-agent-certs\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.760396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.760396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-cnibin\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.760396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-slash\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.760396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-conf-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.760860 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-bin\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.760965 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt9j\" (UniqueName: \"kubernetes.io/projected/93793e48-633c-432e-b5e7-8eeba7006e71-kube-api-access-vjt9j\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.760965 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.760965 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-modprobe-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.761109 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.760983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-kubelet-config\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.761109 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.761109 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93793e48-633c-432e-b5e7-8eeba7006e71-ovn-node-metrics-cert\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.761109 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.761109 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm2w\" (UniqueName: \"kubernetes.io/projected/c0d84fe3-69b7-4a5c-978d-3362efbb5477-kube-api-access-nhm2w\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwsb\" (UniqueName: \"kubernetes.io/projected/5b44a16f-1b5d-46db-a58f-1ac32c313498-kube-api-access-cvwsb\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-hostroot\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-env-overrides\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwmz\" (UniqueName: \"kubernetes.io/projected/84d73353-8ae9-43ef-929f-c33c67702e6a-kube-api-access-8bwmz\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-tuned\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.761318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-system-cni-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-hosts-file\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-k8s-cni-cncf-io\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-bin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-etc-kubernetes\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b44a16f-1b5d-46db-a58f-1ac32c313498-host-slash\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-netns\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-daemon-config\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-conf\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-kubelet\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.761912 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-multus-certs\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762190 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.761806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-kubelet-config\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:18.762190 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-kubelet\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762190 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.762093    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:18.762190 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.762163    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:19.262142793 +0000 UTC m=+2.981239868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:18.762312 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b44a16f-1b5d-46db-a58f-1ac32c313498-host-slash\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.762343 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-var-lib-cni-bin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762410 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-cni-dir\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762493 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.762493 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-k8s-cni-cncf-io\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-etc-kubernetes\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-netns\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-hostroot\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-system-cni-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.762590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/786b0e59-4b3f-47c7-9c97-cda697025943-serviceca\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.762736 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7vm\" (UniqueName: \"kubernetes.io/projected/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-kube-api-access-gz7vm\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.762736 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-systemd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.762736 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-host\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.762825 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-tmp\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.762825 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-cnibin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762889 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-socket-dir-parent\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.762889 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-kubelet\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.762944 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-log-socket\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.762944 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/786b0e59-4b3f-47c7-9c97-cda697025943-host\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.763001 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc49d\" (UniqueName: \"kubernetes.io/projected/786b0e59-4b3f-47c7-9c97-cda697025943-kube-api-access-sc49d\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.763001 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b44a16f-1b5d-46db-a58f-1ac32c313498-iptables-alerter-script\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.763001 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.762992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/786b0e59-4b3f-47c7-9c97-cda697025943-serviceca\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.763087 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-host-run-multus-certs\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.763087 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-daemon-config\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.763147 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-var-lib-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.763147 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-hosts-file\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.763233 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-netd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.763233 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-cnibin\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.763233 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.763322 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.763322 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8567d\" (UniqueName: \"kubernetes.io/projected/eb3c189b-2662-4686-accc-a20201cd7ef3-kube-api-access-8567d\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.763380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-cni-binary-copy\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.763380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2cf\" (UniqueName: \"kubernetes.io/projected/7633ccec-856e-413c-8d65-3aaeb1201b6a-kube-api-access-7g2cf\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.763479 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-systemd-units\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.763627 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/786b0e59-4b3f-47c7-9c97-cda697025943-host\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.763627 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/932754fd-04f5-4fc3-bd81-36b598b7311a-multus-socket-dir-parent\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.763746 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb3c189b-2662-4686-accc-a20201cd7ef3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.763795 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb3c189b-2662-4686-accc-a20201cd7ef3-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.763795 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b44a16f-1b5d-46db-a58f-1ac32c313498-iptables-alerter-script\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.763910 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.763887    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:18.764497 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.764475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/932754fd-04f5-4fc3-bd81-36b598b7311a-cni-binary-copy\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.766532 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.766512    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:18.766613 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.766538    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:18.766613 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.766551    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:18.766613 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.766610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:19.266596275 +0000 UTC m=+2.985693331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:18.768335 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.768309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c9f51084-955f-4058-b1e2-2da5bcea710b-agent-certs\") pod \"konnectivity-agent-d65wk\" (UID: \"c9f51084-955f-4058-b1e2-2da5bcea710b\") " pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.768954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.768901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8w4q\" (UniqueName: \"kubernetes.io/projected/932754fd-04f5-4fc3-bd81-36b598b7311a-kube-api-access-k8w4q\") pod \"multus-4chnh\" (UID: \"932754fd-04f5-4fc3-bd81-36b598b7311a\") " pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.773682 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.773657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwsb\" (UniqueName: \"kubernetes.io/projected/5b44a16f-1b5d-46db-a58f-1ac32c313498-kube-api-access-cvwsb\") pod \"iptables-alerter-6p5cm\" (UID: \"5b44a16f-1b5d-46db-a58f-1ac32c313498\") " pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.773935 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.773909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8567d\" (UniqueName: \"kubernetes.io/projected/eb3c189b-2662-4686-accc-a20201cd7ef3-kube-api-access-8567d\") pod \"multus-additional-cni-plugins-xmplp\" (UID: \"eb3c189b-2662-4686-accc-a20201cd7ef3\") " pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.774020 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.773996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7vm\" (UniqueName: \"kubernetes.io/projected/4a2385e9-c1ed-48e9-91fd-d3a66af674b3-kube-api-access-gz7vm\") pod \"node-resolver-mh7lg\" (UID: \"4a2385e9-c1ed-48e9-91fd-d3a66af674b3\") " pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.774597 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.774498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc49d\" (UniqueName: \"kubernetes.io/projected/786b0e59-4b3f-47c7-9c97-cda697025943-kube-api-access-sc49d\") pod \"node-ca-fjkkt\" (UID: \"786b0e59-4b3f-47c7-9c97-cda697025943\") " pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.864314 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-var-lib-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864314 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-netd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2cf\" (UniqueName: \"kubernetes.io/projected/7633ccec-856e-413c-8d65-3aaeb1201b6a-kube-api-access-7g2cf\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-systemd-units\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-var-lib-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-netd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-systemd-units\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-etc-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-lib-modules\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-kubernetes\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.864575 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-run\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-netns\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-etc-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-kubernetes\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-script-lib\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-run-netns\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-lib-modules\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-run\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-systemd\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-systemd\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-sys\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-sys\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-ovn\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-node-log\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-config\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-device-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.865154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-ovn\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-node-log\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-device-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysconfig\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-var-lib-kubelet\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-slash\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-bin\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.864995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt9j\" (UniqueName: \"kubernetes.io/projected/93793e48-633c-432e-b5e7-8eeba7006e71-kube-api-access-vjt9j\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-var-lib-kubelet\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-slash\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-cni-bin\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysconfig\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-modprobe-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.865977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93793e48-633c-432e-b5e7-8eeba7006e71-ovn-node-metrics-cert\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-socket-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-openvswitch\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm2w\" (UniqueName: \"kubernetes.io/projected/c0d84fe3-69b7-4a5c-978d-3362efbb5477-kube-api-access-nhm2w\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-registration-dir\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-env-overrides\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-modprobe-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-config\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-ovnkube-script-lib\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwmz\" (UniqueName: \"kubernetes.io/projected/84d73353-8ae9-43ef-929f-c33c67702e6a-kube-api-access-8bwmz\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-tuned\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-conf\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84d73353-8ae9-43ef-929f-c33c67702e6a-sys-fs\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.866611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-systemd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-host\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93793e48-633c-432e-b5e7-8eeba7006e71-env-overrides\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-tmp\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-d\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-kubelet\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-host\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.865688    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-log-socket\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-host-kubelet\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-sysctl-conf\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-run-systemd\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:18.865748    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:19.365732552 +0000 UTC m=+3.084829607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:18.867148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.865760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93793e48-633c-432e-b5e7-8eeba7006e71-log-socket\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.867945 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.867921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-etc-tuned\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.867945 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.867921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93793e48-633c-432e-b5e7-8eeba7006e71-ovn-node-metrics-cert\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.868181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.868159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d84fe3-69b7-4a5c-978d-3362efbb5477-tmp\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.879955 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.879886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm2w\" (UniqueName: \"kubernetes.io/projected/c0d84fe3-69b7-4a5c-978d-3362efbb5477-kube-api-access-nhm2w\") pod \"tuned-ktv6k\" (UID: \"c0d84fe3-69b7-4a5c-978d-3362efbb5477\") " pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:18.881645 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.881618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwmz\" (UniqueName: \"kubernetes.io/projected/84d73353-8ae9-43ef-929f-c33c67702e6a-kube-api-access-8bwmz\") pod \"aws-ebs-csi-driver-node-9ll8h\" (UID: \"84d73353-8ae9-43ef-929f-c33c67702e6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.882839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.882815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2cf\" (UniqueName: \"kubernetes.io/projected/7633ccec-856e-413c-8d65-3aaeb1201b6a-kube-api-access-7g2cf\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:18.883146 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.883126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt9j\" (UniqueName: \"kubernetes.io/projected/93793e48-633c-432e-b5e7-8eeba7006e71-kube-api-access-vjt9j\") pod \"ovnkube-node-tn8vm\" (UID: \"93793e48-633c-432e-b5e7-8eeba7006e71\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.949054 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.949025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p5cm"
Sep 16 18:40:18.955420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.955400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mh7lg"
Sep 16 18:40:18.964647 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.964617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4chnh"
Sep 16 18:40:18.969413 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.969385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmplp"
Sep 16 18:40:18.975633 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.975612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:18.981906 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.981890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fjkkt"
Sep 16 18:40:18.988574 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.988558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:18.994618 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.994598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h"
Sep 16 18:40:18.999895 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:18.999874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktv6k"
Sep 16 18:40:19.268129 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.268058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:19.268129 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.268099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:19.268340 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268213    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:19.268340 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268252    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:19.268340 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268266    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:20.268250864 +0000 UTC m=+3.987347918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:19.268340 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268275    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:19.268340 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268293    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:19.268607 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.268345    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:20.268327564 +0000 UTC m=+3.987424634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:19.369037 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.369007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:19.369223 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.369137    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:19.369223 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:19.369198    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:20.369181961 +0000 UTC m=+4.088279018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:19.607640 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.607584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786b0e59_4b3f_47c7_9c97_cda697025943.slice/crio-e181b08e8e30e2494b384e5fbba78b4806e2a08259a99a07bd1fecca614d4e95 WatchSource:0}: Error finding container e181b08e8e30e2494b384e5fbba78b4806e2a08259a99a07bd1fecca614d4e95: Status 404 returned error can't find the container with id e181b08e8e30e2494b384e5fbba78b4806e2a08259a99a07bd1fecca614d4e95
Sep 16 18:40:19.609483 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.609365    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d73353_8ae9_43ef_929f_c33c67702e6a.slice/crio-b66fb9128455263a5999b6ecc491b1f92374b43e17490735091ba744c036023c WatchSource:0}: Error finding container b66fb9128455263a5999b6ecc491b1f92374b43e17490735091ba744c036023c: Status 404 returned error can't find the container with id b66fb9128455263a5999b6ecc491b1f92374b43e17490735091ba744c036023c
Sep 16 18:40:19.616497 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.612874    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3c189b_2662_4686_accc_a20201cd7ef3.slice/crio-15efee30118baf7783c005fc6faed4421b27acf3be8bdc274cbb1d7d4ccb0b5b WatchSource:0}: Error finding container 15efee30118baf7783c005fc6faed4421b27acf3be8bdc274cbb1d7d4ccb0b5b: Status 404 returned error can't find the container with id 15efee30118baf7783c005fc6faed4421b27acf3be8bdc274cbb1d7d4ccb0b5b
Sep 16 18:40:19.616497 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.613753    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b44a16f_1b5d_46db_a58f_1ac32c313498.slice/crio-73644f4a85fc2f0cb555e7bd34ee26676142802f4f10e256e8a6a53f59c5ed36 WatchSource:0}: Error finding container 73644f4a85fc2f0cb555e7bd34ee26676142802f4f10e256e8a6a53f59c5ed36: Status 404 returned error can't find the container with id 73644f4a85fc2f0cb555e7bd34ee26676142802f4f10e256e8a6a53f59c5ed36
Sep 16 18:40:19.616497 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.615332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f51084_955f_4058_b1e2_2da5bcea710b.slice/crio-c3d3086f2fca3a1aca9713694416a94473cf6d65640abf52ee35e6d064aeb6ea WatchSource:0}: Error finding container c3d3086f2fca3a1aca9713694416a94473cf6d65640abf52ee35e6d064aeb6ea: Status 404 returned error can't find the container with id c3d3086f2fca3a1aca9713694416a94473cf6d65640abf52ee35e6d064aeb6ea
Sep 16 18:40:19.616497 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.616277    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d84fe3_69b7_4a5c_978d_3362efbb5477.slice/crio-99d0e1d14890ddbf946d55c882cccacb1dba497054192cde43a4be7f0915dbbb WatchSource:0}: Error finding container 99d0e1d14890ddbf946d55c882cccacb1dba497054192cde43a4be7f0915dbbb: Status 404 returned error can't find the container with id 99d0e1d14890ddbf946d55c882cccacb1dba497054192cde43a4be7f0915dbbb
Sep 16 18:40:19.617606 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.617126    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2385e9_c1ed_48e9_91fd_d3a66af674b3.slice/crio-d38eb1ee9cde3a483d79292a259e6f9e1b8f485972c6aa1795002127458caf67 WatchSource:0}: Error finding container d38eb1ee9cde3a483d79292a259e6f9e1b8f485972c6aa1795002127458caf67: Status 404 returned error can't find the container with id d38eb1ee9cde3a483d79292a259e6f9e1b8f485972c6aa1795002127458caf67
Sep 16 18:40:19.618310 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.618269    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93793e48_633c_432e_b5e7_8eeba7006e71.slice/crio-0119df1ff5dccd4e607906ff5be206d6e0bad95ee8d95ede498aabfee995f6fe WatchSource:0}: Error finding container 0119df1ff5dccd4e607906ff5be206d6e0bad95ee8d95ede498aabfee995f6fe: Status 404 returned error can't find the container with id 0119df1ff5dccd4e607906ff5be206d6e0bad95ee8d95ede498aabfee995f6fe
Sep 16 18:40:19.620133 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:19.620078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932754fd_04f5_4fc3_bd81_36b598b7311a.slice/crio-b805fd00a2dacfbad5ca7389d026930b95bc774ca52d7c155aadbda56075f63a WatchSource:0}: Error finding container b805fd00a2dacfbad5ca7389d026930b95bc774ca52d7c155aadbda56075f63a: Status 404 returned error can't find the container with id b805fd00a2dacfbad5ca7389d026930b95bc774ca52d7c155aadbda56075f63a
Sep 16 18:40:19.692055 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.692005    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:17 +0000 UTC" deadline="2028-03-22 01:53:25.660086276 +0000 UTC"
Sep 16 18:40:19.692055 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.692050    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13255h13m5.968039232s"
Sep 16 18:40:19.753587 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.753552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal" event={"ID":"2d64174e840dde57cd70ce29f4e8fd34","Type":"ContainerStarted","Data":"f6d2458e68c947957a9cce5e3758781a8904ddfb4a687f0666da68661462c202"}
Sep 16 18:40:19.754825 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.754795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4chnh" event={"ID":"932754fd-04f5-4fc3-bd81-36b598b7311a","Type":"ContainerStarted","Data":"b805fd00a2dacfbad5ca7389d026930b95bc774ca52d7c155aadbda56075f63a"}
Sep 16 18:40:19.755977 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.755953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"0119df1ff5dccd4e607906ff5be206d6e0bad95ee8d95ede498aabfee995f6fe"}
Sep 16 18:40:19.756947 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.756926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mh7lg" event={"ID":"4a2385e9-c1ed-48e9-91fd-d3a66af674b3","Type":"ContainerStarted","Data":"d38eb1ee9cde3a483d79292a259e6f9e1b8f485972c6aa1795002127458caf67"}
Sep 16 18:40:19.757987 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.757969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktv6k" event={"ID":"c0d84fe3-69b7-4a5c-978d-3362efbb5477","Type":"ContainerStarted","Data":"99d0e1d14890ddbf946d55c882cccacb1dba497054192cde43a4be7f0915dbbb"}
Sep 16 18:40:19.758948 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.758920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d65wk" event={"ID":"c9f51084-955f-4058-b1e2-2da5bcea710b","Type":"ContainerStarted","Data":"c3d3086f2fca3a1aca9713694416a94473cf6d65640abf52ee35e6d064aeb6ea"}
Sep 16 18:40:19.759847 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.759829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p5cm" event={"ID":"5b44a16f-1b5d-46db-a58f-1ac32c313498","Type":"ContainerStarted","Data":"73644f4a85fc2f0cb555e7bd34ee26676142802f4f10e256e8a6a53f59c5ed36"}
Sep 16 18:40:19.760700 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.760684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjkkt" event={"ID":"786b0e59-4b3f-47c7-9c97-cda697025943","Type":"ContainerStarted","Data":"e181b08e8e30e2494b384e5fbba78b4806e2a08259a99a07bd1fecca614d4e95"}
Sep 16 18:40:19.761611 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.761596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerStarted","Data":"15efee30118baf7783c005fc6faed4421b27acf3be8bdc274cbb1d7d4ccb0b5b"}
Sep 16 18:40:19.762621 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:19.762603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h" event={"ID":"84d73353-8ae9-43ef-929f-c33c67702e6a","Type":"ContainerStarted","Data":"b66fb9128455263a5999b6ecc491b1f92374b43e17490735091ba744c036023c"}
Sep 16 18:40:20.275754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.275430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:20.275754 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275692    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:20.275754 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275722    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:20.275754 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275736    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:20.275754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.275737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:20.276136 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.275776655 +0000 UTC m=+5.994873727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:20.276136 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275842    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:20.276136 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.275923    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.27590879 +0000 UTC m=+5.995005858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:20.376800 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.376767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:20.376977 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.376901    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:20.376977 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.376949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.376937265 +0000 UTC m=+6.096034334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:20.643177 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.642812    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.744581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.744711    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.745094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.745186    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.745285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:20.745413 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:20.745377    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:20.785625 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.784592    2579 generic.go:356] "Generic (PLEG): container finished" podID="d621e84069d8ebb64c644982a8593730" containerID="031c3dc622ca13d612ed5afcb69d8f12751f471b4cd771633f0b9378b96733b1" exitCode=0
Sep 16 18:40:20.785625 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.785422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal" event={"ID":"d621e84069d8ebb64c644982a8593730","Type":"ContainerDied","Data":"031c3dc622ca13d612ed5afcb69d8f12751f471b4cd771633f0b9378b96733b1"}
Sep 16 18:40:20.801648 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:20.800330    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-81.ec2.internal" podStartSLOduration=3.800309567 podStartE2EDuration="3.800309567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:19.768220068 +0000 UTC m=+3.487317144" watchObservedRunningTime="2026-09-16 18:40:20.800309567 +0000 UTC m=+4.519406647"
Sep 16 18:40:21.803526 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:21.803486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal" event={"ID":"d621e84069d8ebb64c644982a8593730","Type":"ContainerStarted","Data":"a80169c788a70ce095b6fee5309a100f1d1808783167c502105b52bbcfd72f4c"}
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.290992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.291085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291220    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291245    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291257    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291314    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:26.291295288 +0000 UTC m=+10.010392350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291782    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.291913 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.291831    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:26.291817424 +0000 UTC m=+10.010914483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.392129 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.392068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:22.392330 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.392312    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.392420 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.392381    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:26.392363063 +0000 UTC m=+10.111460125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.746280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.746403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.746807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.746913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:22.746968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:22.747216 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:22.747038    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:24.744623 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:24.744589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:24.745046 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:24.744652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:24.745046 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:24.744667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:24.745046 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:24.744771    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:24.745202 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:24.745060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:24.745202 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:24.745163    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:26.325478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.325429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:26.325882 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.325584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:26.325882 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.325720    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:26.325882 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.325734    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:26.325882 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.325743    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:26.325882 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.325783    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.325770117 +0000 UTC m=+18.044867173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:26.326118 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.326096    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:26.326178 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.326143    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.326129395 +0000 UTC m=+18.045226454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:26.426513 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.426480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:26.426697 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.426619    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:26.426697 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.426675    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.426658028 +0000 UTC m=+18.145755083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:26.745111 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.745026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:26.745310 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.745142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:26.745551 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.745531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:26.745824 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.745642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:26.745824 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:26.745704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:26.745824 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:26.745784    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:28.743565 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:28.743529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:28.743565 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:28.743551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:28.744060 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:28.743570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:28.744060 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:28.743661    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:28.744060 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:28.743718    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:28.744060 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:28.743809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:30.744432 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:30.744345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:30.744432 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:30.744345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:30.744908 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:30.744357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:30.744908 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:30.744532    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:30.744908 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:30.744500    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:30.744908 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:30.744616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:32.743747 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:32.743712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:32.744116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:32.743712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:32.744116 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:32.743829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:32.744116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:32.743719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:32.744116 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:32.743901    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:32.744116 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:32.743958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:34.381853 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.381810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.381913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.381980    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.382034    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.382049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.38203253 +0000 UTC m=+34.101129608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.382054    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.382069    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.382331 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.382117    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.382102273 +0000 UTC m=+34.101199334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.482380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.482339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:34.482579 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.482512    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.482579 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.482570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.482551633 +0000 UTC m=+34.201648701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.744276 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.744198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:34.744276 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.744222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:34.744276 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:34.744278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:34.744570 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.744407    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:34.744570 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.744498    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:34.744689 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:34.744588    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:36.744115 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:36.744083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:36.744572 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:36.744190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:36.744572 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:36.744274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:36.744572 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:36.744365    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:36.744572 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:36.744413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:36.744572 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:36.744491    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:37.836974 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.836944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerStarted","Data":"3f6e3d421e5185bb8e9b640d6192544cbc5ec180c5e96f623a12bbe5fef602b7"}
Sep 16 18:40:37.839051 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.839027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h" event={"ID":"84d73353-8ae9-43ef-929f-c33c67702e6a","Type":"ContainerStarted","Data":"ec2986bf82d766add6f5967c2bb18216400e541d9837a96bcd8703dcf37fe56e"}
Sep 16 18:40:37.840621 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.840599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d65wk" event={"ID":"c9f51084-955f-4058-b1e2-2da5bcea710b","Type":"ContainerStarted","Data":"8e1f4a34664bd67817696de57f44a2d08a34932e41961ee80c1d187fe2c47909"}
Sep 16 18:40:37.842417 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.842397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fjkkt" event={"ID":"786b0e59-4b3f-47c7-9c97-cda697025943","Type":"ContainerStarted","Data":"c8a467d650f8857ea79be5fee6e433d6bdf468880226ea4d8149fddc474ebd95"}
Sep 16 18:40:37.860574 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.860521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-81.ec2.internal" podStartSLOduration=20.860508651 podStartE2EDuration="20.860508651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:21.82224044 +0000 UTC m=+5.541337520" watchObservedRunningTime="2026-09-16 18:40:37.860508651 +0000 UTC m=+21.579605728"
Sep 16 18:40:37.880014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.879966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d65wk" podStartSLOduration=3.951143128 podStartE2EDuration="21.879951627s" totalImagesPullingTime="17.928808499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:37.879823833 +0000 UTC m=+21.598920909" watchObservedRunningTime="2026-09-16 18:40:37.879951627 +0000 UTC m=+21.599048704"
Sep 16 18:40:37.897535 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:37.897485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fjkkt" podStartSLOduration=3.961337921 podStartE2EDuration="21.897451185s" totalImagesPullingTime="17.936113264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:37.897340918 +0000 UTC m=+21.616438016" watchObservedRunningTime="2026-09-16 18:40:37.897451185 +0000 UTC m=+21.616548263"
Sep 16 18:40:38.744398 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.744230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:38.744548 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.744231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:38.744548 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:38.744506    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:38.744623 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.744297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:38.744623 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:38.744559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:38.744683 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:38.744664    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:38.849159 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.848973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849563    2579 generic.go:356] "Generic (PLEG): container finished" podID="93793e48-633c-432e-b5e7-8eeba7006e71" containerID="9a0f13af763c867cc0c18b7874f02010d03ce878caa21efb9a34cbda481c4679" exitCode=1
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"e29fb1ac04f43ff1c442ef08e10b79f2b8be311b900242eb3be9a58b18f51fb8"}
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"a5c06d8aee0b5fc92b89ad1a47a7f9986630923026fd17fffef5473266cfb069"}
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"3fd3069b60e0345aedd60f50eb7f7fca2874d8a5e3fc6a9f576b430de80e1cad"}
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"a6768d653c106231e1f88ccbe961abc6a4d8fd2aff748707dfc83e43c311c8d3"}
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerDied","Data":"9a0f13af763c867cc0c18b7874f02010d03ce878caa21efb9a34cbda481c4679"}
Sep 16 18:40:38.849968 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.849899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"fdafde4c86c54d9ac2da525c43f8afb359bfc1fc92294cadcd117148f4bc68f1"}
Sep 16 18:40:38.851186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.851136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mh7lg" event={"ID":"4a2385e9-c1ed-48e9-91fd-d3a66af674b3","Type":"ContainerStarted","Data":"749ef8222b9f12ceb0775687e65416f56e11cb8ccb371238f4409dbc6ad270d6"}
Sep 16 18:40:38.852860 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.852840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktv6k" event={"ID":"c0d84fe3-69b7-4a5c-978d-3362efbb5477","Type":"ContainerStarted","Data":"53aafa4f1a5b2407e729715eece3c964122bfeb380c3a1c7f7d346b9143d7f8a"}
Sep 16 18:40:38.854983 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.854962    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="3f6e3d421e5185bb8e9b640d6192544cbc5ec180c5e96f623a12bbe5fef602b7" exitCode=0
Sep 16 18:40:38.855084 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.855032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"3f6e3d421e5185bb8e9b640d6192544cbc5ec180c5e96f623a12bbe5fef602b7"}
Sep 16 18:40:38.857283 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.857261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4chnh" event={"ID":"932754fd-04f5-4fc3-bd81-36b598b7311a","Type":"ContainerStarted","Data":"579b34493225061cf59ea31ce0f571cc2cc31a1fecf447732c7a747ad13a9946"}
Sep 16 18:40:38.869169 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.869129    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mh7lg" podStartSLOduration=4.849318863 podStartE2EDuration="22.869117259s" totalImagesPullingTime="18.019798396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.868850293 +0000 UTC m=+22.587947371" watchObservedRunningTime="2026-09-16 18:40:38.869117259 +0000 UTC m=+22.588214336"
Sep 16 18:40:38.914973 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.914868    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4chnh" podStartSLOduration=4.622909619 podStartE2EDuration="22.914854334s" totalImagesPullingTime="18.291944715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.914666941 +0000 UTC m=+22.633764018" watchObservedRunningTime="2026-09-16 18:40:38.914854334 +0000 UTC m=+22.633951411"
Sep 16 18:40:38.932177 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:38.932126    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ktv6k" podStartSLOduration=4.91108071 podStartE2EDuration="22.932109083s" totalImagesPullingTime="18.021028373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.931412587 +0000 UTC m=+22.650509664" watchObservedRunningTime="2026-09-16 18:40:38.932109083 +0000 UTC m=+22.651206161"
Sep 16 18:40:39.326300 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.326277    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:39.721162 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.720994    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:39.326294571Z","UUID":"6df33d6f-4dd4-4ac6-ab84-5b1ad741a9d4","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:39.723216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.723192    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:39.723216 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.723220    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:39.861344 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.861308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p5cm" event={"ID":"5b44a16f-1b5d-46db-a58f-1ac32c313498","Type":"ContainerStarted","Data":"6a8f84d50a8d4e90605bc511310adbea6f93d136f3eada9cfa176f8e9304c999"}
Sep 16 18:40:39.864340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:39.864267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h" event={"ID":"84d73353-8ae9-43ef-929f-c33c67702e6a","Type":"ContainerStarted","Data":"12ce547cd8c67e9962a4d3c1b5999962cb3c90858ea4f85d6a96afdf68496bad"}
Sep 16 18:40:40.744322 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.744293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:40.744491 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:40.744405    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:40.744491 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.744294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:40.744613 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:40.744554    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:40.744664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.744641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:40.744772 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:40.744744    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:40.868917 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.868882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h" event={"ID":"84d73353-8ae9-43ef-929f-c33c67702e6a","Type":"ContainerStarted","Data":"620a9a9085b6a34fc1091638ad52ba266b121b2b115487046a9f5183ae0c2dd4"}
Sep 16 18:40:40.892173 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.892120    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6p5cm" podStartSLOduration=6.870017312 podStartE2EDuration="24.892097336s" totalImagesPullingTime="18.022080024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.8784802 +0000 UTC m=+23.597577277" watchObservedRunningTime="2026-09-16 18:40:40.892097336 +0000 UTC m=+24.611194416"
Sep 16 18:40:40.892564 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:40.892526    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ll8h" podStartSLOduration=4.379752617 podStartE2EDuration="24.892512124s" totalImagesPullingTime="20.512759507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:40.891646996 +0000 UTC m=+24.610744073" watchObservedRunningTime="2026-09-16 18:40:40.892512124 +0000 UTC m=+24.611609202"
Sep 16 18:40:41.094492 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.094395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:41.095083 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.095049    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:41.874191 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.874162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:40:41.874674 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.874639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"54df6ea50447e7785df1ace99d43457603385f3cdd3aeaa5c54d0dfebfcea793"}
Sep 16 18:40:41.875393 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.875356    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:41.875781 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:41.875765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d65wk"
Sep 16 18:40:42.743810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:42.743747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:42.743966 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:42.743747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:42.743966 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:42.743883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:42.743966 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:42.743876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:42.744138 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:42.744001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:42.744138 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:42.744079    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:43.882419 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.882256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:40:43.883153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.882816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"8c6ac675c03390780f90bd78b4fe487ea495095550206a0bc768f5d14a2d377f"}
Sep 16 18:40:43.883153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.883030    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:43.883153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.883093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:43.883283 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.883224    2579 scope.go:117] "RemoveContainer" containerID="9a0f13af763c867cc0c18b7874f02010d03ce878caa21efb9a34cbda481c4679"
Sep 16 18:40:43.885056 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.885019    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="1d55e2371cf597d46762a754a8200ba63028d9af2a4718b3fef543dff3418075" exitCode=0
Sep 16 18:40:43.885230 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.885100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"1d55e2371cf597d46762a754a8200ba63028d9af2a4718b3fef543dff3418075"}
Sep 16 18:40:43.900473 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:43.900432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:44.743321 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.743292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:44.743631 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.743300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:44.743631 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:44.743407    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:44.743631 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:44.743508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:44.743631 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.743299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:44.743631 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:44.743586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:44.890022 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.890002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:40:44.890418 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.890323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" event={"ID":"93793e48-633c-432e-b5e7-8eeba7006e71","Type":"ContainerStarted","Data":"92096b5d262b4d59f0d27e48f9adbed14966645fdca9318dcd4333aae3a3971e"}
Sep 16 18:40:44.890672 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.890655    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:44.906904 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.906885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:40:44.928578 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:44.928537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm" podStartSLOduration=10.426694136 podStartE2EDuration="28.928521451s" totalImagesPullingTime="18.501827315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:44.927096658 +0000 UTC m=+28.646193746" watchObservedRunningTime="2026-09-16 18:40:44.928521451 +0000 UTC m=+28.647618527"
Sep 16 18:40:45.243480 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.243262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h69vp"]
Sep 16 18:40:45.243647 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.243550    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:45.243687 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:45.243650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:45.246215 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.246189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cdlgj"]
Sep 16 18:40:45.246313 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.246265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:45.246359 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:45.246339    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:45.247740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.247714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bc7ds"]
Sep 16 18:40:45.248484 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.247861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:45.248484 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:45.247997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:45.895065 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.895035    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="9ff9104898ac807d9b533bba44a9efca1f948f1b8d9886a34aa32d18c78165e8" exitCode=0
Sep 16 18:40:45.895429 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:45.895117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"9ff9104898ac807d9b533bba44a9efca1f948f1b8d9886a34aa32d18c78165e8"}
Sep 16 18:40:46.744068 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:46.744040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:46.744195 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:46.744132    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:46.744195 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:46.744157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:46.744195 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:46.744175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:46.744353 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:46.744271    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:46.744412 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:46.744347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:46.899980 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:46.899953    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="67fa168d06b0d60aa8fbc2706c6b7140c641674eafea2809e6f38cc786964801" exitCode=0
Sep 16 18:40:46.900484 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:46.900035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"67fa168d06b0d60aa8fbc2706c6b7140c641674eafea2809e6f38cc786964801"}
Sep 16 18:40:48.743837 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:48.743804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:48.744279 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:48.743805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:48.744279 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:48.743934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:48.744279 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:48.743948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:48.744279 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:48.744062    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:48.744279 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:48.744121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:50.403138 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.402879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403046    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403265    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret podName:ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:22.403244353 +0000 UTC m=+66.122341428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret") pod "global-pull-secret-syncer-h69vp" (UID: "ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.403199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403297    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403315    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403327    2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzkqt for pod openshift-network-diagnostics/network-check-target-cdlgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.403649 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.403390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt podName:1c88996b-5555-47e0-91a8-618176e2d3d7 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:22.403374456 +0000 UTC m=+66.122471527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzkqt" (UniqueName: "kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt") pod "network-check-target-cdlgj" (UID: "1c88996b-5555-47e0-91a8-618176e2d3d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.504647 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.504611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:50.504806 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.504745    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:50.504864 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.504808    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:22.504787681 +0000 UTC m=+66.223884737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:50.743858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.743777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:50.743858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.743799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:50.744080 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:50.743777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:50.744080 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.743920    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h69vp" podUID="ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2"
Sep 16 18:40:50.744080 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.744004    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc7ds" podUID="7633ccec-856e-413c-8d65-3aaeb1201b6a"
Sep 16 18:40:50.744225 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:50.744112    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdlgj" podUID="1c88996b-5555-47e0-91a8-618176e2d3d7"
Sep 16 18:40:51.591503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.591403    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-81.ec2.internal" event="NodeReady"
Sep 16 18:40:51.591831 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.591553    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:51.665336 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.665308    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvvb2"]
Sep 16 18:40:51.667420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.667390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.668164 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.668142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fn88h"]
Sep 16 18:40:51.669867 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.669854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:51.670898 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.670881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:51.671269 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.671250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f92fp\""
Sep 16 18:40:51.671324 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.671250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:51.672948 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.672925    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-nkrcn\" is forbidden: User \"system:node:ip-10-0-136-81.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-136-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkrcn\"" type="*v1.Secret"
Sep 16 18:40:51.673320 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.673296    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-136-81.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-136-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret"
Sep 16 18:40:51.673387 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.673348    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-136-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 18:40:51.673387 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.673348    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-136-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 18:40:51.685135 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.685112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn88h"]
Sep 16 18:40:51.685652 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.685633    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvvb2"]
Sep 16 18:40:51.814671 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77b5c39-4844-45ed-afcf-779b2f3b0145-tmp-dir\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.814671 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:51.814869 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzch\" (UniqueName: \"kubernetes.io/projected/ccc6de4b-def5-4625-b397-0791b85b12b2-kube-api-access-6pzch\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:51.814869 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.814869 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77b5c39-4844-45ed-afcf-779b2f3b0145-config-volume\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.814869 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.814771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkxt\" (UniqueName: \"kubernetes.io/projected/c77b5c39-4844-45ed-afcf-779b2f3b0145-kube-api-access-xvkxt\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.914562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.914482    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="98b6b2557798e72cff2f99c6fb656f340462458e54f5aa5d37d7034594124247" exitCode=0
Sep 16 18:40:51.914562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.914547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"98b6b2557798e72cff2f99c6fb656f340462458e54f5aa5d37d7034594124247"}
Sep 16 18:40:51.915058 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:51.915181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzch\" (UniqueName: \"kubernetes.io/projected/ccc6de4b-def5-4625-b397-0791b85b12b2-kube-api-access-6pzch\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:51.915181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.915181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77b5c39-4844-45ed-afcf-779b2f3b0145-config-volume\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.915181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkxt\" (UniqueName: \"kubernetes.io/projected/c77b5c39-4844-45ed-afcf-779b2f3b0145-kube-api-access-xvkxt\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.915727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77b5c39-4844-45ed-afcf-779b2f3b0145-tmp-dir\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.915727 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.915248    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:51.915727 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:51.915311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.415293716 +0000 UTC m=+36.134390784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:51.915727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77b5c39-4844-45ed-afcf-779b2f3b0145-tmp-dir\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.915863 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.915776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77b5c39-4844-45ed-afcf-779b2f3b0145-config-volume\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:51.928857 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:51.928835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkxt\" (UniqueName: \"kubernetes.io/projected/c77b5c39-4844-45ed-afcf-779b2f3b0145-kube-api-access-xvkxt\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:52.419534 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.419489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:52.419678 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:52.419649    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:52.419729 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:52.419714    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:53.419698915 +0000 UTC m=+37.138795970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:52.642010 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.641978    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:52.743396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.743320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:40:52.743543 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.743319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:40:52.743543 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.743489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:40:52.746498 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.746478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mffp6\""
Sep 16 18:40:52.746623 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.746559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:52.747364 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.747339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:52.747606 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.747588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-md6w5\""
Sep 16 18:40:52.747725 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.747623    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:52.747725 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.747642    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:52.850926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.850903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkrcn\""
Sep 16 18:40:52.915627 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:52.915597    2579 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:40:52.915772 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:52.915670    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert podName:ccc6de4b-def5-4625-b397-0791b85b12b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:53.415650704 +0000 UTC m=+37.134747774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert") pod "ingress-canary-fn88h" (UID: "ccc6de4b-def5-4625-b397-0791b85b12b2") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:40:52.920219 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.920193    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb3c189b-2662-4686-accc-a20201cd7ef3" containerID="7837e76a42fa56d55d10d2ba121512b91e8ece2fcaaf488b2b25a53952bb38f4" exitCode=0
Sep 16 18:40:52.920324 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:52.920236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerDied","Data":"7837e76a42fa56d55d10d2ba121512b91e8ece2fcaaf488b2b25a53952bb38f4"}
Sep 16 18:40:53.191599 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.191569    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:53.208120 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.208097    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:53.219023 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.218994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzch\" (UniqueName: \"kubernetes.io/projected/ccc6de4b-def5-4625-b397-0791b85b12b2-kube-api-access-6pzch\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:53.428347 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.428313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:53.428529 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.428380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:53.428529 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:53.428485    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:53.428652 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:53.428547    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:55.428532558 +0000 UTC m=+39.147629613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:53.430801 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.430782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc6de4b-def5-4625-b397-0791b85b12b2-cert\") pod \"ingress-canary-fn88h\" (UID: \"ccc6de4b-def5-4625-b397-0791b85b12b2\") " pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:53.482045 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.482019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn88h"
Sep 16 18:40:53.792621 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.792383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn88h"]
Sep 16 18:40:53.796053 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:40:53.796030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc6de4b_def5_4625_b397_0791b85b12b2.slice/crio-c52ef67acbe7a1c822e0813e31e3fbadf9d72ba71784a0830790679131f57c55 WatchSource:0}: Error finding container c52ef67acbe7a1c822e0813e31e3fbadf9d72ba71784a0830790679131f57c55: Status 404 returned error can't find the container with id c52ef67acbe7a1c822e0813e31e3fbadf9d72ba71784a0830790679131f57c55
Sep 16 18:40:53.925949 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.925920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmplp" event={"ID":"eb3c189b-2662-4686-accc-a20201cd7ef3","Type":"ContainerStarted","Data":"fbd90468915ce2d0a05fa586921acb2d13ba107e531432e0f0493e5c55393736"}
Sep 16 18:40:53.927096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.927071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn88h" event={"ID":"ccc6de4b-def5-4625-b397-0791b85b12b2","Type":"ContainerStarted","Data":"c52ef67acbe7a1c822e0813e31e3fbadf9d72ba71784a0830790679131f57c55"}
Sep 16 18:40:53.949793 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:53.949750    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xmplp" podStartSLOduration=10.044058555 podStartE2EDuration="37.949736121s" totalImagesPullingTime="27.905677566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.948493316 +0000 UTC m=+37.667590392" watchObservedRunningTime="2026-09-16 18:40:53.949736121 +0000 UTC m=+37.668833192"
Sep 16 18:40:55.444721 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:55.444682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:55.445146 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:55.444880    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:55.445146 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:55.444960    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.444937445 +0000 UTC m=+43.164034512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:56.935510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:56.935447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn88h" event={"ID":"ccc6de4b-def5-4625-b397-0791b85b12b2","Type":"ContainerStarted","Data":"73d721d8f04f9c83b73e671876f3dead787d272c3b290df78a7ea86e9814b086"}
Sep 16 18:40:56.952591 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:56.952549    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fn88h" podStartSLOduration=3.6736172480000002 podStartE2EDuration="5.952534688s" totalImagesPullingTime="2.27891744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:56.952281442 +0000 UTC m=+40.671378520" watchObservedRunningTime="2026-09-16 18:40:56.952534688 +0000 UTC m=+40.671631756"
Sep 16 18:40:59.472479 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:40:59.472423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:40:59.472868 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:59.472577    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.472868 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:40:59.472642    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:07.472625494 +0000 UTC m=+51.191722552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:07.530698 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:07.530657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:07.531246 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:07.530836    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:07.531246 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:07.530920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:23.530897733 +0000 UTC m=+67.249994792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:16.911374 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:16.911342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn8vm"
Sep 16 18:41:22.428102 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.428059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:41:22.428570 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.428116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:41:22.431277 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.431247    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:22.431423 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.431389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:22.441186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.441163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:22.441662 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.441644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2-original-pull-secret\") pod \"global-pull-secret-syncer-h69vp\" (UID: \"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2\") " pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:41:22.451617 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.451590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkqt\" (UniqueName: \"kubernetes.io/projected/1c88996b-5555-47e0-91a8-618176e2d3d7-kube-api-access-nzkqt\") pod \"network-check-target-cdlgj\" (UID: \"1c88996b-5555-47e0-91a8-618176e2d3d7\") " pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:41:22.464125 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.464102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mffp6\""
Sep 16 18:41:22.466149 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.466135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h69vp"
Sep 16 18:41:22.472267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.472252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:41:22.529021 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.528991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:41:22.531597 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.531577    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:22.540286 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:22.540001    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:22.540286 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:22.540086    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs podName:7633ccec-856e-413c-8d65-3aaeb1201b6a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.540062534 +0000 UTC m=+130.259159591 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs") pod "network-metrics-daemon-bc7ds" (UID: "7633ccec-856e-413c-8d65-3aaeb1201b6a") : secret "metrics-daemon-secret" not found
Sep 16 18:41:22.771934 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.771857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h69vp"]
Sep 16 18:41:22.775028 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:22.775003    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1836ce_5a61_4d27_b9fc_7ca2c7d36cf2.slice/crio-4eee34670194eea812bae75be0e41575c94ec8e6cfddad5d3f5d2a18d23457a2 WatchSource:0}: Error finding container 4eee34670194eea812bae75be0e41575c94ec8e6cfddad5d3f5d2a18d23457a2: Status 404 returned error can't find the container with id 4eee34670194eea812bae75be0e41575c94ec8e6cfddad5d3f5d2a18d23457a2
Sep 16 18:41:22.800849 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.800825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cdlgj"]
Sep 16 18:41:22.803840 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:22.803819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c88996b_5555_47e0_91a8_618176e2d3d7.slice/crio-0044a7a77057e6f8a5294d9d2002db78be874730b480de255491a0e6118df564 WatchSource:0}: Error finding container 0044a7a77057e6f8a5294d9d2002db78be874730b480de255491a0e6118df564: Status 404 returned error can't find the container with id 0044a7a77057e6f8a5294d9d2002db78be874730b480de255491a0e6118df564
Sep 16 18:41:22.997261 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.997223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cdlgj" event={"ID":"1c88996b-5555-47e0-91a8-618176e2d3d7","Type":"ContainerStarted","Data":"0044a7a77057e6f8a5294d9d2002db78be874730b480de255491a0e6118df564"}
Sep 16 18:41:22.998307 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:22.998288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h69vp" event={"ID":"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2","Type":"ContainerStarted","Data":"4eee34670194eea812bae75be0e41575c94ec8e6cfddad5d3f5d2a18d23457a2"}
Sep 16 18:41:23.535549 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:23.535511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:23.536211 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:23.535685    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:23.536211 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:23.535759    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls podName:c77b5c39-4844-45ed-afcf-779b2f3b0145 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:55.53573695 +0000 UTC m=+99.254834019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls") pod "dns-default-pvvb2" (UID: "c77b5c39-4844-45ed-afcf-779b2f3b0145") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:28.013201 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:28.013160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cdlgj" event={"ID":"1c88996b-5555-47e0-91a8-618176e2d3d7","Type":"ContainerStarted","Data":"febf9ce77c1e6e0f766edc0f993367ccc23e2200817a8831c926157137707732"}
Sep 16 18:41:28.013725 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:28.013241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:41:28.014665 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:28.014647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h69vp" event={"ID":"ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2","Type":"ContainerStarted","Data":"d1b824c70634ffcac075d5a993d2a53d42de0f6f70ddf1444b7e0030efdfc373"}
Sep 16 18:41:28.030353 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:28.030309    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cdlgj" podStartSLOduration=67.578891798 podStartE2EDuration="1m12.030296522s" totalImagesPullingTime="4.451404724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:28.02999978 +0000 UTC m=+71.749096856" watchObservedRunningTime="2026-09-16 18:41:28.030296522 +0000 UTC m=+71.749393599"
Sep 16 18:41:28.049141 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:28.049097    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h69vp" podStartSLOduration=67.565744039 podStartE2EDuration="1m12.049084108s" totalImagesPullingTime="4.483340069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:28.047952329 +0000 UTC m=+71.767049407" watchObservedRunningTime="2026-09-16 18:41:28.049084108 +0000 UTC m=+71.768181184"
Sep 16 18:41:37.214562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.214525    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-qsnj6"]
Sep 16 18:41:37.217119 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.217102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.219742 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.219721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:37.219868 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.219738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:41:37.220277 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.220262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:37.221242 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.221219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dbhbb\""
Sep 16 18:41:37.221316 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.221281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:41:37.226447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.226428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:41:37.231353 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-tmp\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.231476 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-service-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.231476 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg44h\" (UniqueName: \"kubernetes.io/projected/ff45e963-083a-410b-a861-7f5379c427ac-kube-api-access-jg44h\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.231593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.231593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-snapshots\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.231593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.231562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff45e963-083a-410b-a861-7f5379c427ac-serving-cert\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.242168 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.242145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qsnj6"]
Sep 16 18:41:37.318573 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.318540    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"]
Sep 16 18:41:37.321224 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.321207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"
Sep 16 18:41:37.323405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.323377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"]
Sep 16 18:41:37.324755 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.324736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9n7gc\""
Sep 16 18:41:37.325171 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.325153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:37.325239 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.325228    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:37.326527 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.326510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:41:37.326651 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.326633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.329066 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.329049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.331823 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-snapshots\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.331900 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff45e963-083a-410b-a861-7f5379c427ac-serving-cert\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.331900 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-tmp\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.331900 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-service-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.331900 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg44h\" (UniqueName: \"kubernetes.io/projected/ff45e963-083a-410b-a861-7f5379c427ac-kube-api-access-jg44h\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.332092 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglf6\" (UniqueName: \"kubernetes.io/projected/c95fdaad-b588-4818-b2eb-1ca0eece2d31-kube-api-access-hglf6\") pod \"volume-data-source-validator-86695b7cc7-pppw2\" (UID: \"c95fdaad-b588-4818-b2eb-1ca0eece2d31\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"
Sep 16 18:41:37.332092 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.331967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.332437 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.332417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-tmp\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.332538 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.332443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ff45e963-083a-410b-a861-7f5379c427ac-snapshots\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.332658 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.332614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-service-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.332830 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.332778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff45e963-083a-410b-a861-7f5379c427ac-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.334365 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.334347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff45e963-083a-410b-a861-7f5379c427ac-serving-cert\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.362511 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.362480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:41:37.362625 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.362491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:37.362625 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.362519    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:37.362625 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.362619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:37.362804 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.362791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:37.372761 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.372734    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jkf2p\""
Sep 16 18:41:37.373014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.372993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w8crq\""
Sep 16 18:41:37.373169 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.373153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:37.376050 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.376026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"]
Sep 16 18:41:37.376927 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.376907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"]
Sep 16 18:41:37.402886 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.402853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:41:37.404491 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.404472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg44h\" (UniqueName: \"kubernetes.io/projected/ff45e963-083a-410b-a861-7f5379c427ac-kube-api-access-jg44h\") pod \"insights-operator-5844b856db-qsnj6\" (UID: \"ff45e963-083a-410b-a861-7f5379c427ac\") " pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.410076 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.410058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:37.433010 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.432983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.433096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgdt\" (UniqueName: \"kubernetes.io/projected/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-kube-api-access-7sgdt\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.433096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433096 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433225 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433225 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433286 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433286 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433286 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtwt\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.433367 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.433291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglf6\" (UniqueName: \"kubernetes.io/projected/c95fdaad-b588-4818-b2eb-1ca0eece2d31-kube-api-access-hglf6\") pod \"volume-data-source-validator-86695b7cc7-pppw2\" (UID: \"c95fdaad-b588-4818-b2eb-1ca0eece2d31\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"
Sep 16 18:41:37.436043 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.436025    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"]
Sep 16 18:41:37.439676 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.439662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.443417 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.443400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:41:37.443544 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.443420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:37.443544 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.443508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:41:37.443984 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.443964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bwnqh\""
Sep 16 18:41:37.449868 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.449853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:37.469845 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.469796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglf6\" (UniqueName: \"kubernetes.io/projected/c95fdaad-b588-4818-b2eb-1ca0eece2d31-kube-api-access-hglf6\") pod \"volume-data-source-validator-86695b7cc7-pppw2\" (UID: \"c95fdaad-b588-4818-b2eb-1ca0eece2d31\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"
Sep 16 18:41:37.473344 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.473319    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"]
Sep 16 18:41:37.526223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.526200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qsnj6"
Sep 16 18:41:37.534181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534291 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534291 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534594 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534664 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:37.534529    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:37.534664 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:37.534634    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffd4b88cd-pnj85: secret "image-registry-tls" not found
Sep 16 18:41:37.534664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.534664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.534911 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:37.534707    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls podName:7ec644e7-103c-4d0b-9403-50219832b385 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:38.034683456 +0000 UTC m=+81.753780564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls") pod "image-registry-5ffd4b88cd-pnj85" (UID: "7ec644e7-103c-4d0b-9403-50219832b385") : secret "image-registry-tls" not found
Sep 16 18:41:37.534911 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69j7\" (UniqueName: \"kubernetes.io/projected/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-kube-api-access-k69j7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.534911 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtwt\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.534911 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.535112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.534984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.535112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.535039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgdt\" (UniqueName: \"kubernetes.io/projected/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-kube-api-access-7sgdt\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.535112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.535067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.535419 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:37.535291    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:37.535419 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:37.535365    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls podName:3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:38.035347848 +0000 UTC m=+81.754444918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g9bkt" (UID: "3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d") : secret "samples-operator-tls" not found
Sep 16 18:41:37.535419 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.535375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.535627 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.535488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.537053 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.537029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.537239 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.537219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.551812 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.551790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtwt\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.551948 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.551927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgdt\" (UniqueName: \"kubernetes.io/projected/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-kube-api-access-7sgdt\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:37.552765 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.552734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:37.630113 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.630081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"
Sep 16 18:41:37.635542 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.635505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.635636 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.635548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.635697 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.635676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69j7\" (UniqueName: \"kubernetes.io/projected/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-kube-api-access-k69j7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.636164 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.636142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.637828 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.637812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.649862 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.649841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69j7\" (UniqueName: \"kubernetes.io/projected/d0db4211-6f85-4e65-9cb7-bd14ae7f782f-kube-api-access-k69j7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lqzxm\" (UID: \"d0db4211-6f85-4e65-9cb7-bd14ae7f782f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.748279 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.748205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"
Sep 16 18:41:37.807610 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.807586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qsnj6"]
Sep 16 18:41:37.811003 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:37.810976    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff45e963_083a_410b_a861_7f5379c427ac.slice/crio-df21849bea1a452bec6d5ddd1075c257a577c93b520873941bc8aa87bb0d9f58 WatchSource:0}: Error finding container df21849bea1a452bec6d5ddd1075c257a577c93b520873941bc8aa87bb0d9f58: Status 404 returned error can't find the container with id df21849bea1a452bec6d5ddd1075c257a577c93b520873941bc8aa87bb0d9f58
Sep 16 18:41:37.908507 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:37.908481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2"]
Sep 16 18:41:37.911119 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:37.911088    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95fdaad_b588_4818_b2eb_1ca0eece2d31.slice/crio-5f0222d36f6afb453b35c3981660d3235d6565d5d275407ebe25063eb9b594cd WatchSource:0}: Error finding container 5f0222d36f6afb453b35c3981660d3235d6565d5d275407ebe25063eb9b594cd: Status 404 returned error can't find the container with id 5f0222d36f6afb453b35c3981660d3235d6565d5d275407ebe25063eb9b594cd
Sep 16 18:41:38.024094 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.024026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm"]
Sep 16 18:41:38.026929 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:38.026900    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0db4211_6f85_4e65_9cb7_bd14ae7f782f.slice/crio-8fdb50fe10876e8d4a9993d0f01f7a3b8bc45d843a2fd3ea165681ae62630d8e WatchSource:0}: Error finding container 8fdb50fe10876e8d4a9993d0f01f7a3b8bc45d843a2fd3ea165681ae62630d8e: Status 404 returned error can't find the container with id 8fdb50fe10876e8d4a9993d0f01f7a3b8bc45d843a2fd3ea165681ae62630d8e
Sep 16 18:41:38.037679 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.037655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:38.037766 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.037714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:38.037818 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:38.037806    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:38.037867 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:38.037819    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffd4b88cd-pnj85: secret "image-registry-tls" not found
Sep 16 18:41:38.037867 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:38.037831    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:38.037934 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:38.037868    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls podName:7ec644e7-103c-4d0b-9403-50219832b385 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.037851638 +0000 UTC m=+82.756948694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls") pod "image-registry-5ffd4b88cd-pnj85" (UID: "7ec644e7-103c-4d0b-9403-50219832b385") : secret "image-registry-tls" not found
Sep 16 18:41:38.037934 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:38.037882    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls podName:3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.037876243 +0000 UTC m=+82.756973298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g9bkt" (UID: "3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d") : secret "samples-operator-tls" not found
Sep 16 18:41:38.038373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.038353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm" event={"ID":"d0db4211-6f85-4e65-9cb7-bd14ae7f782f","Type":"ContainerStarted","Data":"8fdb50fe10876e8d4a9993d0f01f7a3b8bc45d843a2fd3ea165681ae62630d8e"}
Sep 16 18:41:38.039560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.039541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2" event={"ID":"c95fdaad-b588-4818-b2eb-1ca0eece2d31","Type":"ContainerStarted","Data":"5f0222d36f6afb453b35c3981660d3235d6565d5d275407ebe25063eb9b594cd"}
Sep 16 18:41:38.040803 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:38.040784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qsnj6" event={"ID":"ff45e963-083a-410b-a861-7f5379c427ac","Type":"ContainerStarted","Data":"df21849bea1a452bec6d5ddd1075c257a577c93b520873941bc8aa87bb0d9f58"}
Sep 16 18:41:39.044833 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:39.044805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:39.044878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:39.044955    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:39.044996    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:39.045006    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffd4b88cd-pnj85: secret "image-registry-tls" not found
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:39.045023    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls podName:3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.044995685 +0000 UTC m=+84.764092757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g9bkt" (UID: "3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d") : secret "samples-operator-tls" not found
Sep 16 18:41:39.045304 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:39.045041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls podName:7ec644e7-103c-4d0b-9403-50219832b385 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.045029282 +0000 UTC m=+84.764126339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls") pod "image-registry-5ffd4b88cd-pnj85" (UID: "7ec644e7-103c-4d0b-9403-50219832b385") : secret "image-registry-tls" not found
Sep 16 18:41:39.045670 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:39.045645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2" event={"ID":"c95fdaad-b588-4818-b2eb-1ca0eece2d31","Type":"ContainerStarted","Data":"249ef3613add06a9ab9f4d2a1685a3b364baf9b261a43814d6b82394b943226e"}
Sep 16 18:41:39.070973 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:39.070926    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pppw2" podStartSLOduration=1.086259721 podStartE2EDuration="2.070910687s" totalImagesPullingTime="984.650966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.070219462 +0000 UTC m=+82.789316539" watchObservedRunningTime="2026-09-16 18:41:39.070910687 +0000 UTC m=+82.790007764"
Sep 16 18:41:41.053954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.053911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm" event={"ID":"d0db4211-6f85-4e65-9cb7-bd14ae7f782f","Type":"ContainerStarted","Data":"b0bf1b919678348cef361b180e8b1b1d2d0c47cdfa45871e9e683a66fa474361"}
Sep 16 18:41:41.055572 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.055549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qsnj6" event={"ID":"ff45e963-083a-410b-a861-7f5379c427ac","Type":"ContainerStarted","Data":"1c933ef7d812b853cca8549982c556a189f57d4073963241bb53ea3a4a79c8e7"}
Sep 16 18:41:41.060394 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.060366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:41.060525 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.060444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:41.060576 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.060529    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:41.060576 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.060538    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:41.060576 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.060544    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffd4b88cd-pnj85: secret "image-registry-tls" not found
Sep 16 18:41:41.060678 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.060596    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls podName:7ec644e7-103c-4d0b-9403-50219832b385 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.060579611 +0000 UTC m=+88.779676666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls") pod "image-registry-5ffd4b88cd-pnj85" (UID: "7ec644e7-103c-4d0b-9403-50219832b385") : secret "image-registry-tls" not found
Sep 16 18:41:41.060678 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.060610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls podName:3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.060602661 +0000 UTC m=+88.779699716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g9bkt" (UID: "3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d") : secret "samples-operator-tls" not found
Sep 16 18:41:41.078155 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.078110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm" podStartSLOduration=1.9549564579999998 podStartE2EDuration="4.07809337s" totalImagesPullingTime="2.123136912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:41.077252307 +0000 UTC m=+84.796349383" watchObservedRunningTime="2026-09-16 18:41:41.07809337 +0000 UTC m=+84.797190450"
Sep 16 18:41:41.096805 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.096760    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-qsnj6" podStartSLOduration=1.7608965589999999 podStartE2EDuration="4.096744209s" totalImagesPullingTime="2.33584765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:41.094987668 +0000 UTC m=+84.814084756" watchObservedRunningTime="2026-09-16 18:41:41.096744209 +0000 UTC m=+84.815841287"
Sep 16 18:41:41.432151 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.432084    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n2tjk"]
Sep 16 18:41:41.435534 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.435519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.438562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.438538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:41.438703 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.438591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8f7h6\""
Sep 16 18:41:41.438848 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.438829    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:41.460375 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.460355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2tjk"]
Sep 16 18:41:41.463308 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.463283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.463405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.463358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx92\" (UniqueName: \"kubernetes.io/projected/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-api-access-hzx92\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.463481 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.463421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-data-volume\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.463534 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.463502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.463585 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.463550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-crio-socket\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.564909 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.564863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565107 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.564937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-crio-socket\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565107 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.564996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565107 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.565060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx92\" (UniqueName: \"kubernetes.io/projected/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-api-access-hzx92\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565107 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.564997    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.565355 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.565124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-data-volume\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565355 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.565144    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.065124813 +0000 UTC m=+85.784221868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.565355 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.565038    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.565355 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.565068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-crio-socket\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.565355 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:41.565226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.065208393 +0000 UTC m=+85.784305450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.565686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.565489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-data-volume\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:41.589506 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:41.589474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx92\" (UniqueName: \"kubernetes.io/projected/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-api-access-hzx92\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:42.069686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:42.069655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:42.070022 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:42.069706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:42.070022 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:42.069769    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.070022 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:42.069785    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.070022 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:42.069814    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.06980306 +0000 UTC m=+86.788900128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.070022 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:42.069833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:43.069819843 +0000 UTC m=+86.788916898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.079166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:43.079133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:43.079556 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:43.079228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:43.079556 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:43.079252    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:43.079556 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:43.079321    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.079302791 +0000 UTC m=+88.798399860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:43.079556 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:43.079322    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.079556 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:43.079358    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.079348067 +0000 UTC m=+88.798445123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.179132 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:43.179102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mh7lg_4a2385e9-c1ed-48e9-91fd-d3a66af674b3/dns-node-resolver/0.log"
Sep 16 18:41:44.178840 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:44.178817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjkkt_786b0e59-4b3f-47c7-9c97-cda697025943/node-ca/0.log"
Sep 16 18:41:45.094957 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:45.094926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:45.094957 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:45.094960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:45.095008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:45.095025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095077    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095093    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095116    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095131    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffd4b88cd-pnj85: secret "image-registry-tls" not found
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.095122338 +0000 UTC m=+92.814219406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:45.095151 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095147    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls podName:3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:53.095141662 +0000 UTC m=+96.814238717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g9bkt" (UID: "3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d") : secret "samples-operator-tls" not found
Sep 16 18:41:45.095410 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095165    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:45.095410 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls podName:7ec644e7-103c-4d0b-9403-50219832b385 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:53.095154769 +0000 UTC m=+96.814251838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls") pod "image-registry-5ffd4b88cd-pnj85" (UID: "7ec644e7-103c-4d0b-9403-50219832b385") : secret "image-registry-tls" not found
Sep 16 18:41:45.095410 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:45.095196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.095185065 +0000 UTC m=+92.814282135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:45.378626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:45.378543    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn88h_ccc6de4b-def5-4625-b397-0791b85b12b2/serve-healthcheck-canary/0.log"
Sep 16 18:41:49.125791 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:49.125762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:49.126250 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:49.125820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:49.126250 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:49.125915    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.126250 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:49.125929    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:49.126250 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:49.125970    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:57.125955205 +0000 UTC m=+100.845052280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.126250 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:41:49.125997    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm podName:7d82e923-85db-4b4f-bc3c-9991af8ee5cd nodeName:}" failed. No retries permitted until 2026-09-16 18:41:57.125982148 +0000 UTC m=+100.845079216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2tjk" (UID: "7d82e923-85db-4b4f-bc3c-9991af8ee5cd") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:53.160092 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.160060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:53.160492 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.160117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:53.162608 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.162583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g9bkt\" (UID: \"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:53.162608 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.162604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"image-registry-5ffd4b88cd-pnj85\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") " pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:53.242122 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.242073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"
Sep 16 18:41:53.245892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.245868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:53.529448 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.529331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt"]
Sep 16 18:41:53.538258 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:53.538236    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:41:53.551493 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:53.551438    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec644e7_103c_4d0b_9403_50219832b385.slice/crio-f0bce2e20b08d233d63cae24b7b3f1f5701aebc246314eca5dfb5c37f090078a WatchSource:0}: Error finding container f0bce2e20b08d233d63cae24b7b3f1f5701aebc246314eca5dfb5c37f090078a: Status 404 returned error can't find the container with id f0bce2e20b08d233d63cae24b7b3f1f5701aebc246314eca5dfb5c37f090078a
Sep 16 18:41:54.096378 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:54.096342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" event={"ID":"7ec644e7-103c-4d0b-9403-50219832b385","Type":"ContainerStarted","Data":"b410be50ad71ee08c269e941d6359c9c1cf82ed72af8626b62c3942be3cc46a5"}
Sep 16 18:41:54.096582 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:54.096441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:41:54.096582 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:54.096476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" event={"ID":"7ec644e7-103c-4d0b-9403-50219832b385","Type":"ContainerStarted","Data":"f0bce2e20b08d233d63cae24b7b3f1f5701aebc246314eca5dfb5c37f090078a"}
Sep 16 18:41:54.097755 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:54.097714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt" event={"ID":"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d","Type":"ContainerStarted","Data":"f7fbc4b1be5f5585d8a4a140d34600b6d1ea87feb672290cf8c571db02211980"}
Sep 16 18:41:54.119496 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:54.119432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" podStartSLOduration=17.119417047 podStartE2EDuration="17.119417047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.118859179 +0000 UTC m=+97.837956257" watchObservedRunningTime="2026-09-16 18:41:54.119417047 +0000 UTC m=+97.838514123"
Sep 16 18:41:55.103499 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:55.103429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt" event={"ID":"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d","Type":"ContainerStarted","Data":"996a60c623c9d4623462451d55d99a2b30d319076d9782af3071b0eceb60923f"}
Sep 16 18:41:55.582690 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:55.582654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:55.585168 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:55.585142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77b5c39-4844-45ed-afcf-779b2f3b0145-metrics-tls\") pod \"dns-default-pvvb2\" (UID: \"c77b5c39-4844-45ed-afcf-779b2f3b0145\") " pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:55.880307 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:55.880234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f92fp\""
Sep 16 18:41:55.888220 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:55.888201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:56.107972 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:56.107938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt" event={"ID":"3cb5f21b-b9d9-4f3b-8915-e2d287df7b3d","Type":"ContainerStarted","Data":"c607effab54a38c5691ace10dc312188792f64174e2113a18e61d840fca60fba"}
Sep 16 18:41:56.134872 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:56.134782    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g9bkt" podStartSLOduration=17.718596946 podStartE2EDuration="19.134765425s" totalImagesPullingTime="1.416168479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:56.133721118 +0000 UTC m=+99.852818195" watchObservedRunningTime="2026-09-16 18:41:56.134765425 +0000 UTC m=+99.853862502"
Sep 16 18:41:56.186393 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:56.186361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvvb2"]
Sep 16 18:41:56.189614 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:56.189590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77b5c39_4844_45ed_afcf_779b2f3b0145.slice/crio-5ecc22d3ccf458c6adc8bf1081701f0782d798c8ba55b3e93c564870097e953c WatchSource:0}: Error finding container 5ecc22d3ccf458c6adc8bf1081701f0782d798c8ba55b3e93c564870097e953c: Status 404 returned error can't find the container with id 5ecc22d3ccf458c6adc8bf1081701f0782d798c8ba55b3e93c564870097e953c
Sep 16 18:41:57.113475 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.113416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvvb2" event={"ID":"c77b5c39-4844-45ed-afcf-779b2f3b0145","Type":"ContainerStarted","Data":"5ecc22d3ccf458c6adc8bf1081701f0782d798c8ba55b3e93c564870097e953c"}
Sep 16 18:41:57.195476 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.195411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:57.195796 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.195773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:57.196070 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.196025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:57.198812 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.198788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d82e923-85db-4b4f-bc3c-9991af8ee5cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2tjk\" (UID: \"7d82e923-85db-4b4f-bc3c-9991af8ee5cd\") " pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:57.344748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.344716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2tjk"
Sep 16 18:41:57.646035 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:57.645964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2tjk"]
Sep 16 18:41:57.649191 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:41:57.649166    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82e923_85db_4b4f_bc3c_9991af8ee5cd.slice/crio-a929cc7a231ae52a57c5deb307c951ea46a5773d730862f289ab896d0b026ab5 WatchSource:0}: Error finding container a929cc7a231ae52a57c5deb307c951ea46a5773d730862f289ab896d0b026ab5: Status 404 returned error can't find the container with id a929cc7a231ae52a57c5deb307c951ea46a5773d730862f289ab896d0b026ab5
Sep 16 18:41:58.118294 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:58.118253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2tjk" event={"ID":"7d82e923-85db-4b4f-bc3c-9991af8ee5cd","Type":"ContainerStarted","Data":"bbc436c30bddf32da9f1970387b26dca7ca1d85dc726ab25b8f98b259a4e26e5"}
Sep 16 18:41:58.118728 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:58.118325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2tjk" event={"ID":"7d82e923-85db-4b4f-bc3c-9991af8ee5cd","Type":"ContainerStarted","Data":"a929cc7a231ae52a57c5deb307c951ea46a5773d730862f289ab896d0b026ab5"}
Sep 16 18:41:59.019664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.019638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cdlgj"
Sep 16 18:41:59.123119 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.123087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvvb2" event={"ID":"c77b5c39-4844-45ed-afcf-779b2f3b0145","Type":"ContainerStarted","Data":"cde164f0c2fbef33e52c40552d45c8ba80e498bb701913421e6527e8fe53c04e"}
Sep 16 18:41:59.123596 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.123145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvvb2" event={"ID":"c77b5c39-4844-45ed-afcf-779b2f3b0145","Type":"ContainerStarted","Data":"10d74562f391263b12bb526b7cbc35e26c600e214c5f95170dcb9d6bb93c59af"}
Sep 16 18:41:59.123596 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.123189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:41:59.125120 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.125097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2tjk" event={"ID":"7d82e923-85db-4b4f-bc3c-9991af8ee5cd","Type":"ContainerStarted","Data":"823a2c53f5b6b4330dd01e38501b4826dcfde5e6cd8eb56fce95155b83bea140"}
Sep 16 18:41:59.142630 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:41:59.142589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pvvb2" podStartSLOduration=65.634339867 podStartE2EDuration="1m8.142575021s" totalImagesPullingTime="2.508235154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:59.14207302 +0000 UTC m=+102.861170121" watchObservedRunningTime="2026-09-16 18:41:59.142575021 +0000 UTC m=+102.861672098"
Sep 16 18:42:01.137817 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:01.137784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2tjk" event={"ID":"7d82e923-85db-4b4f-bc3c-9991af8ee5cd","Type":"ContainerStarted","Data":"2721665d34a3e6497eff2aa2e6eb9c98cbbcff31e5bb827fabe9f2aba9045bd8"}
Sep 16 18:42:01.163951 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:01.163909    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n2tjk" podStartSLOduration=17.770348225 podStartE2EDuration="20.163895293s" totalImagesPullingTime="2.393547068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:01.163478737 +0000 UTC m=+104.882575812" watchObservedRunningTime="2026-09-16 18:42:01.163895293 +0000 UTC m=+104.882992413"
Sep 16 18:42:08.073060 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.073028    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"]
Sep 16 18:42:08.075351 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.075329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.078088 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.078068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:08.079371 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.079350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:42:08.079607 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.079589    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:08.079682 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.079621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:08.092602 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.092584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"]
Sep 16 18:42:08.113690 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.113669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:42:08.118045 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.118021    2579 patch_prober.go:28] interesting pod/image-registry-5ffd4b88cd-pnj85 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:08.118117 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.118060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" podUID="7ec644e7-103c-4d0b-9403-50219832b385" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:08.174773 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.174747    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"]
Sep 16 18:42:08.176605 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.176590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.179143 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.179122    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-cw9ft"]
Sep 16 18:42:08.179268 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.179250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:42:08.179315 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.179277    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:42:08.179363 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.179346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tcllz\""
Sep 16 18:42:08.180683 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.180663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d0777a-d03d-437a-beea-d4be0343ca7d-tmp\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.180794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.180759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71d0777a-d03d-437a-beea-d4be0343ca7d-klusterlet-config\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.180863 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.180809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crcwm\" (UniqueName: \"kubernetes.io/projected/71d0777a-d03d-437a-beea-d4be0343ca7d-kube-api-access-crcwm\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.181212 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.181198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:08.183402 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.183385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f894ffccc-qzt2k"]
Sep 16 18:42:08.184138 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.184117    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:08.184232 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.184196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:08.184272 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.184244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wcgwj\""
Sep 16 18:42:08.185552 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.185537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.201339 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.201318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"]
Sep 16 18:42:08.233482 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.233433    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-cw9ft"]
Sep 16 18:42:08.234770 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.234752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f894ffccc-qzt2k"]
Sep 16 18:42:08.281154 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71d0777a-d03d-437a-beea-d4be0343ca7d-klusterlet-config\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.281299 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crcwm\" (UniqueName: \"kubernetes.io/projected/71d0777a-d03d-437a-beea-d4be0343ca7d-kube-api-access-crcwm\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.281299 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-installation-pull-secrets\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281299 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwvl\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-kube-api-access-lhwvl\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281299 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-tls\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281299 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-trusted-ca\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d0777a-d03d-437a-beea-d4be0343ca7d-tmp\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90ab2308-69cd-46b1-98e9-bfea1334b3f0-ca-trust-extracted\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-certificates\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-image-registry-private-configuration\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-bound-sa-token\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.281600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.282071 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.282071 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhjg\" (UniqueName: \"kubernetes.io/projected/015dd585-fed3-4d0b-b652-64f27f0d000c-kube-api-access-cnhjg\") pod \"downloads-76f99584f4-cw9ft\" (UID: \"015dd585-fed3-4d0b-b652-64f27f0d000c\") " pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:08.282071 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.281765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d0777a-d03d-437a-beea-d4be0343ca7d-tmp\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.283865 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.283847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/71d0777a-d03d-437a-beea-d4be0343ca7d-klusterlet-config\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.292542 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.292523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crcwm\" (UniqueName: \"kubernetes.io/projected/71d0777a-d03d-437a-beea-d4be0343ca7d-kube-api-access-crcwm\") pod \"klusterlet-addon-workmgr-9959f7767-rwwjs\" (UID: \"71d0777a-d03d-437a-beea-d4be0343ca7d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.382181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-installation-pull-secrets\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwvl\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-kube-api-access-lhwvl\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-tls\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-trusted-ca\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90ab2308-69cd-46b1-98e9-bfea1334b3f0-ca-trust-extracted\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-certificates\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-image-registry-private-configuration\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-bound-sa-token\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.382510 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.382783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.382783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhjg\" (UniqueName: \"kubernetes.io/projected/015dd585-fed3-4d0b-b652-64f27f0d000c-kube-api-access-cnhjg\") pod \"downloads-76f99584f4-cw9ft\" (UID: \"015dd585-fed3-4d0b-b652-64f27f0d000c\") " pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:08.382783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.382732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90ab2308-69cd-46b1-98e9-bfea1334b3f0-ca-trust-extracted\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.383238 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.383211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-certificates\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.383540 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.383363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.383540 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.383505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90ab2308-69cd-46b1-98e9-bfea1334b3f0-trusted-ca\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.383661 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.383622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:08.385038 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.385016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-image-registry-private-configuration\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.385249 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.385231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90af4f64-52de-4dd6-8ce7-7cfb0022fb05-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-qvssf\" (UID: \"90af4f64-52de-4dd6-8ce7-7cfb0022fb05\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.385495 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.385330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90ab2308-69cd-46b1-98e9-bfea1334b3f0-installation-pull-secrets\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.385567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.385549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-registry-tls\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.400089 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.400069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhjg\" (UniqueName: \"kubernetes.io/projected/015dd585-fed3-4d0b-b652-64f27f0d000c-kube-api-access-cnhjg\") pod \"downloads-76f99584f4-cw9ft\" (UID: \"015dd585-fed3-4d0b-b652-64f27f0d000c\") " pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:08.400554 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.400526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-bound-sa-token\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.400846 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.400830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwvl\" (UniqueName: \"kubernetes.io/projected/90ab2308-69cd-46b1-98e9-bfea1334b3f0-kube-api-access-lhwvl\") pod \"image-registry-7f894ffccc-qzt2k\" (UID: \"90ab2308-69cd-46b1-98e9-bfea1334b3f0\") " pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.486709 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.486682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"
Sep 16 18:42:08.492430 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.492414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:08.497938 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.497904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:08.659691 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.659608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"]
Sep 16 18:42:08.663329 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:08.663302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0777a_d03d_437a_beea_d4be0343ca7d.slice/crio-0d88d434e6d836f6e63d937208bda8ddc2f636fd16940f9628aefdc48b51301d WatchSource:0}: Error finding container 0d88d434e6d836f6e63d937208bda8ddc2f636fd16940f9628aefdc48b51301d: Status 404 returned error can't find the container with id 0d88d434e6d836f6e63d937208bda8ddc2f636fd16940f9628aefdc48b51301d
Sep 16 18:42:08.765674 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.765646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf"]
Sep 16 18:42:08.768473 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:08.768437    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90af4f64_52de_4dd6_8ce7_7cfb0022fb05.slice/crio-035ad251cd27f4446c749d437b7a1b71e377d916b02b856d178e4515cada755d WatchSource:0}: Error finding container 035ad251cd27f4446c749d437b7a1b71e377d916b02b856d178e4515cada755d: Status 404 returned error can't find the container with id 035ad251cd27f4446c749d437b7a1b71e377d916b02b856d178e4515cada755d
Sep 16 18:42:08.775597 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.775574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-cw9ft"]
Sep 16 18:42:08.777035 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:08.777015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f894ffccc-qzt2k"]
Sep 16 18:42:08.777924 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:08.777903    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015dd585_fed3_4d0b_b652_64f27f0d000c.slice/crio-d1a04b562f108af4cc7dd41b15071892b0aa66f8d789d8b1e4657ed502e06f94 WatchSource:0}: Error finding container d1a04b562f108af4cc7dd41b15071892b0aa66f8d789d8b1e4657ed502e06f94: Status 404 returned error can't find the container with id d1a04b562f108af4cc7dd41b15071892b0aa66f8d789d8b1e4657ed502e06f94
Sep 16 18:42:08.778575 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:08.778553    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ab2308_69cd_46b1_98e9_bfea1334b3f0.slice/crio-7a1d5d984477cc3558ca3f08dc483b0c7045c2ca37a56290b7393f79116b6ac1 WatchSource:0}: Error finding container 7a1d5d984477cc3558ca3f08dc483b0c7045c2ca37a56290b7393f79116b6ac1: Status 404 returned error can't find the container with id 7a1d5d984477cc3558ca3f08dc483b0c7045c2ca37a56290b7393f79116b6ac1
Sep 16 18:42:09.130717 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.130688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvvb2"
Sep 16 18:42:09.171525 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.171489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-cw9ft" event={"ID":"015dd585-fed3-4d0b-b652-64f27f0d000c","Type":"ContainerStarted","Data":"d1a04b562f108af4cc7dd41b15071892b0aa66f8d789d8b1e4657ed502e06f94"}
Sep 16 18:42:09.172793 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.172764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf" event={"ID":"90af4f64-52de-4dd6-8ce7-7cfb0022fb05","Type":"ContainerStarted","Data":"035ad251cd27f4446c749d437b7a1b71e377d916b02b856d178e4515cada755d"}
Sep 16 18:42:09.175231 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.175205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs" event={"ID":"71d0777a-d03d-437a-beea-d4be0343ca7d","Type":"ContainerStarted","Data":"0d88d434e6d836f6e63d937208bda8ddc2f636fd16940f9628aefdc48b51301d"}
Sep 16 18:42:09.176806 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.176784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k" event={"ID":"90ab2308-69cd-46b1-98e9-bfea1334b3f0","Type":"ContainerStarted","Data":"c03003ee17d33ed50b7c3057db5a58464186b7d5b15042ba9c6f5c281003ba50"}
Sep 16 18:42:09.176882 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.176819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k" event={"ID":"90ab2308-69cd-46b1-98e9-bfea1334b3f0","Type":"ContainerStarted","Data":"7a1d5d984477cc3558ca3f08dc483b0c7045c2ca37a56290b7393f79116b6ac1"}
Sep 16 18:42:09.176945 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.176922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:09.199776 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:09.199727    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k" podStartSLOduration=1.199710781 podStartE2EDuration="1.199710781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:09.198798295 +0000 UTC m=+112.917895378" watchObservedRunningTime="2026-09-16 18:42:09.199710781 +0000 UTC m=+112.918807852"
Sep 16 18:42:12.192721 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:12.192686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf" event={"ID":"90af4f64-52de-4dd6-8ce7-7cfb0022fb05","Type":"ContainerStarted","Data":"abd233a83ed6ceeb0f54a9a5c0d7e5063d611d4d1fcfe2da0d5dfbcaae8e99f5"}
Sep 16 18:42:12.211242 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:12.211012    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-qvssf" podStartSLOduration=1.629912142 podStartE2EDuration="4.210996598s" totalImagesPullingTime="2.581084456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:12.210750974 +0000 UTC m=+115.929848052" watchObservedRunningTime="2026-09-16 18:42:12.210996598 +0000 UTC m=+115.930093674"
Sep 16 18:42:13.199140 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:13.199058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs" event={"ID":"71d0777a-d03d-437a-beea-d4be0343ca7d","Type":"ContainerStarted","Data":"75f8789fdebd9ccb4d88ca7504d4a8c8ccb221a72dbbf0fc3e71b2d9547219d5"}
Sep 16 18:42:13.217638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:13.217583    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs" podStartSLOduration=1.021442484 podStartE2EDuration="5.217561152s" totalImagesPullingTime="4.196118668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:13.216721131 +0000 UTC m=+116.935818198" watchObservedRunningTime="2026-09-16 18:42:13.217561152 +0000 UTC m=+116.936658230"
Sep 16 18:42:14.203350 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:14.203313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:14.205499 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:14.205473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9959f7767-rwwjs"
Sep 16 18:42:18.120148 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:18.120079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:42:21.479414 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.479380    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"]
Sep 16 18:42:21.482155 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.482129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.486816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:21.486816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:42:21.486816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6vdl2\""
Sep 16 18:42:21.486816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:21.486816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:21.487209 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:21.487209 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.486951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:42:21.498349 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.498324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"]
Sep 16 18:42:21.514907 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.514871    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pj85s"]
Sep 16 18:42:21.521250 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.521222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.524245 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.524222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:21.524245 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.524234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vh8q\""
Sep 16 18:42:21.524422 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.524294    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:21.525493 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.525473    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:21.597192 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.597192 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9492d0dd-e557-484f-80e5-a21721236846-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.597449 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.597449 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48jj\" (UniqueName: \"kubernetes.io/projected/9492d0dd-e557-484f-80e5-a21721236846-kube-api-access-w48jj\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.597449 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.597641 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.597469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698137 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698137 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48jj\" (UniqueName: \"kubernetes.io/projected/9492d0dd-e557-484f-80e5-a21721236846-kube-api-access-w48jj\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94zq\" (UniqueName: \"kubernetes.io/projected/f2edca00-829c-4dc5-8731-99bce57a60d3-kube-api-access-s94zq\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-sys\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9492d0dd-e557-484f-80e5-a21721236846-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-textfile\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-tls\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-wtmp\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-root\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-metrics-client-ca\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.698754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.698701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.699139 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.699020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9492d0dd-e557-484f-80e5-a21721236846-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.699194 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.699137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.699304 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.699281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9492d0dd-e557-484f-80e5-a21721236846-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.701341 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.701317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.701821 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.701803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9492d0dd-e557-484f-80e5-a21721236846-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.711266 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.711247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48jj\" (UniqueName: \"kubernetes.io/projected/9492d0dd-e557-484f-80e5-a21721236846-kube-api-access-w48jj\") pod \"kube-state-metrics-59fbdd77d-g8krb\" (UID: \"9492d0dd-e557-484f-80e5-a21721236846\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.793231 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.793157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"
Sep 16 18:42:21.799954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.799934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-textfile\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.799969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-tls\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-wtmp\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-root\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-metrics-client-ca\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-root\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94zq\" (UniqueName: \"kubernetes.io/projected/f2edca00-829c-4dc5-8731-99bce57a60d3-kube-api-access-s94zq\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-sys\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-wtmp\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2edca00-829c-4dc5-8731-99bce57a60d3-sys\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800332 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-textfile\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.800905 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.800848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.801137 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.801093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2edca00-829c-4dc5-8731-99bce57a60d3-metrics-client-ca\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.803080 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.803033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-tls\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.803307 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.803283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2edca00-829c-4dc5-8731-99bce57a60d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.813816 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.813791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94zq\" (UniqueName: \"kubernetes.io/projected/f2edca00-829c-4dc5-8731-99bce57a60d3-kube-api-access-s94zq\") pod \"node-exporter-pj85s\" (UID: \"f2edca00-829c-4dc5-8731-99bce57a60d3\") " pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:21.832194 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:21.832170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pj85s"
Sep 16 18:42:22.643674 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.643643    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:22.646496 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.646467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.649278 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.649253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:42:22.650675 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.650656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:42:22.651188 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.651167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2rxhr\""
Sep 16 18:42:22.651344 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.651167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:42:22.651801 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.651598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:42:22.651801 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.651644    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:42:22.651801 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.651717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:42:22.652490 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.652280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:42:22.652490 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.652291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:42:22.655929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.655909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:42:22.672401 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.672374    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:22.811297 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4xl\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-kube-api-access-jg4xl\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-out\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-volume\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.811810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.811794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-web-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.912626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.912626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4xl\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-kube-api-access-jg4xl\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.912626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.912915 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913011 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913011 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.912985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-out\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-volume\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-web-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913306 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913306 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913306 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913786 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.913856 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.913838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d7ec7-6587-43c4-9279-4d97ccac5b85-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.916197 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.916151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-out\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.916961 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.916913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-config-volume\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.917336 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.917275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.917911 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.917888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.918017 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.917888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-tls-assets\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.918483 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.918441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.918565 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.918531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-web-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.918616 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.918586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.918777 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.918758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/826d7ec7-6587-43c4-9279-4d97ccac5b85-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.921436 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.921415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4xl\" (UniqueName: \"kubernetes.io/projected/826d7ec7-6587-43c4-9279-4d97ccac5b85-kube-api-access-jg4xl\") pod \"alertmanager-main-0\" (UID: \"826d7ec7-6587-43c4-9279-4d97ccac5b85\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:22.958884 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:22.958857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:24.603254 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.603222    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"]
Sep 16 18:42:24.606250 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.606223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.609149 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:42:24.609273 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609213    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:42:24.609373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:42:24.609524 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:42:24.609651 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p9lfk\""
Sep 16 18:42:24.609651 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9iuqipa726ndk\""
Sep 16 18:42:24.609766 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.609685    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:42:24.619281 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.619260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"]
Sep 16 18:42:24.730743 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.730705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs84h\" (UniqueName: \"kubernetes.io/projected/b328323d-de90-4291-bb87-8011bba6dbaf-kube-api-access-vs84h\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.730922 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.730763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.730922 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.730791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.730922 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.730821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-grpc-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.731079 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.730948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.731079 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.731024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b328323d-de90-4291-bb87-8011bba6dbaf-metrics-client-ca\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.731079 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.731061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.731209 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.731102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.831802 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.831764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs84h\" (UniqueName: \"kubernetes.io/projected/b328323d-de90-4291-bb87-8011bba6dbaf-kube-api-access-vs84h\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.831802 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.831805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832049 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.831840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832105 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.832041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-grpc-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832155 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.832111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832291 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.832264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b328323d-de90-4291-bb87-8011bba6dbaf-metrics-client-ca\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832444 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.832318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.832444 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.832361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.833294 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.833265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b328323d-de90-4291-bb87-8011bba6dbaf-metrics-client-ca\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835147 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835272 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835837 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.835917 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.835860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b328323d-de90-4291-bb87-8011bba6dbaf-secret-grpc-tls\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.840687 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.840668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs84h\" (UniqueName: \"kubernetes.io/projected/b328323d-de90-4291-bb87-8011bba6dbaf-kube-api-access-vs84h\") pod \"thanos-querier-5ffcfc849c-ghbc9\" (UID: \"b328323d-de90-4291-bb87-8011bba6dbaf\") " pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:24.917340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:24.917270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:25.493667 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:25.493631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2edca00_829c_4dc5_8731_99bce57a60d3.slice/crio-1cdba0402222f4212364bfb9449f2807f0228c62eb27520d2ad00261623498a3 WatchSource:0}: Error finding container 1cdba0402222f4212364bfb9449f2807f0228c62eb27520d2ad00261623498a3: Status 404 returned error can't find the container with id 1cdba0402222f4212364bfb9449f2807f0228c62eb27520d2ad00261623498a3
Sep 16 18:42:25.752190 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:25.752158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb"]
Sep 16 18:42:25.755740 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:25.755706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9492d0dd_e557_484f_80e5_a21721236846.slice/crio-c0485c78d64efbb438a8490620452fd83513e74a34b4c1b31b585c9da8fea8d2 WatchSource:0}: Error finding container c0485c78d64efbb438a8490620452fd83513e74a34b4c1b31b585c9da8fea8d2: Status 404 returned error can't find the container with id c0485c78d64efbb438a8490620452fd83513e74a34b4c1b31b585c9da8fea8d2
Sep 16 18:42:25.757593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:25.757568    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:25.764495 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:25.764445    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826d7ec7_6587_43c4_9279_4d97ccac5b85.slice/crio-907ed4f8beed06134378e89d83674bf414505ae1338deffab2339b40be62bccc WatchSource:0}: Error finding container 907ed4f8beed06134378e89d83674bf414505ae1338deffab2339b40be62bccc: Status 404 returned error can't find the container with id 907ed4f8beed06134378e89d83674bf414505ae1338deffab2339b40be62bccc
Sep 16 18:42:25.791287 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:25.791260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"]
Sep 16 18:42:25.792975 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:25.792945    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb328323d_de90_4291_bb87_8011bba6dbaf.slice/crio-f1701f01554923ab529a6f4fafd96a69dd250c449cca2cb2828c12683cf3c510 WatchSource:0}: Error finding container f1701f01554923ab529a6f4fafd96a69dd250c449cca2cb2828c12683cf3c510: Status 404 returned error can't find the container with id f1701f01554923ab529a6f4fafd96a69dd250c449cca2cb2828c12683cf3c510
Sep 16 18:42:26.251425 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.251381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb" event={"ID":"9492d0dd-e557-484f-80e5-a21721236846","Type":"ContainerStarted","Data":"c0485c78d64efbb438a8490620452fd83513e74a34b4c1b31b585c9da8fea8d2"}
Sep 16 18:42:26.254956 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.254848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"f1701f01554923ab529a6f4fafd96a69dd250c449cca2cb2828c12683cf3c510"}
Sep 16 18:42:26.258040 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.257939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"907ed4f8beed06134378e89d83674bf414505ae1338deffab2339b40be62bccc"}
Sep 16 18:42:26.262028 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.261953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj85s" event={"ID":"f2edca00-829c-4dc5-8731-99bce57a60d3","Type":"ContainerStarted","Data":"1cdba0402222f4212364bfb9449f2807f0228c62eb27520d2ad00261623498a3"}
Sep 16 18:42:26.266057 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.265005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-cw9ft" event={"ID":"015dd585-fed3-4d0b-b652-64f27f0d000c","Type":"ContainerStarted","Data":"baaad214f42de016dc2dc337b8355e6df56f628d5fb788769c8edf11e4fa2204"}
Sep 16 18:42:26.266057 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.266005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:26.285724 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.285359    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-cw9ft" podStartSLOduration=1.5050180819999999 podStartE2EDuration="18.285341172s" totalImagesPullingTime="16.78032309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:26.284421623 +0000 UTC m=+130.003518723" watchObservedRunningTime="2026-09-16 18:42:26.285341172 +0000 UTC m=+130.004438249"
Sep 16 18:42:26.285724 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.285550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-cw9ft"
Sep 16 18:42:26.550323 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.550230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:42:26.567580 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.567422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7633ccec-856e-413c-8d65-3aaeb1201b6a-metrics-certs\") pod \"network-metrics-daemon-bc7ds\" (UID: \"7633ccec-856e-413c-8d65-3aaeb1201b6a\") " pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:42:26.656720 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.656499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-md6w5\""
Sep 16 18:42:26.664399 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:26.664151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc7ds"
Sep 16 18:42:27.138686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:27.138558    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bc7ds"]
Sep 16 18:42:27.731758 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:42:27.731725    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7633ccec_856e_413c_8d65_3aaeb1201b6a.slice/crio-b8bee9b6e18990ee0d4b21d58ecf6b23ab7b578bde712d1f49932f153358e199 WatchSource:0}: Error finding container b8bee9b6e18990ee0d4b21d58ecf6b23ab7b578bde712d1f49932f153358e199: Status 404 returned error can't find the container with id b8bee9b6e18990ee0d4b21d58ecf6b23ab7b578bde712d1f49932f153358e199
Sep 16 18:42:28.279993 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:28.279948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc7ds" event={"ID":"7633ccec-856e-413c-8d65-3aaeb1201b6a","Type":"ContainerStarted","Data":"b8bee9b6e18990ee0d4b21d58ecf6b23ab7b578bde712d1f49932f153358e199"}
Sep 16 18:42:30.187376 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.187345    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f894ffccc-qzt2k"
Sep 16 18:42:30.291362 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.291323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb" event={"ID":"9492d0dd-e557-484f-80e5-a21721236846","Type":"ContainerStarted","Data":"5e6bf1c93c6dd686bf6683f596c6590090b155e1f719bc0dcb573e6e21b9ee80"}
Sep 16 18:42:30.291581 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.291405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb" event={"ID":"9492d0dd-e557-484f-80e5-a21721236846","Type":"ContainerStarted","Data":"b604d05e8f9ac734faceb3bb039da71e1c7dc92432f741eccfadddf35f15b99d"}
Sep 16 18:42:30.291581 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.291436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb" event={"ID":"9492d0dd-e557-484f-80e5-a21721236846","Type":"ContainerStarted","Data":"85c674147acf172f1d53dcf2dd542ebfd41c8e80839d4599d7ca45771817bb49"}
Sep 16 18:42:30.293967 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.293923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"72910cc7aee5cd77cc2fef8def0015793a5c7c019a0550157340d88bc0f1af6b"}
Sep 16 18:42:30.294073 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.294001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"394824c42e7a1a4967f33ebb3bd31f07825b3e04bc859b610e909162678ec1aa"}
Sep 16 18:42:30.294073 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.294040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"2325ef22ccb34acf239b9d3bc87158049c76da5a29f7714bd46f62e679abf1af"}
Sep 16 18:42:30.295767 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.295741    2579 generic.go:356] "Generic (PLEG): container finished" podID="826d7ec7-6587-43c4-9279-4d97ccac5b85" containerID="b0fb5080860f23ffac246ea22ff65c52eac2fa936b1ec9ce43e573313ad73912" exitCode=0
Sep 16 18:42:30.296437 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.296416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerDied","Data":"b0fb5080860f23ffac246ea22ff65c52eac2fa936b1ec9ce43e573313ad73912"}
Sep 16 18:42:30.298656 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.298609    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2edca00-829c-4dc5-8731-99bce57a60d3" containerID="7ab3a0608199c8d6069692682592056cbd6ef34cec71e26cc54bbe71c54a6c3e" exitCode=0
Sep 16 18:42:30.298790 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.298655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj85s" event={"ID":"f2edca00-829c-4dc5-8731-99bce57a60d3","Type":"ContainerDied","Data":"7ab3a0608199c8d6069692682592056cbd6ef34cec71e26cc54bbe71c54a6c3e"}
Sep 16 18:42:30.346308 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:30.346232    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-g8krb" podStartSLOduration=5.686770859 podStartE2EDuration="9.346213784s" totalImagesPullingTime="3.659442925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:30.318327724 +0000 UTC m=+134.037424812" watchObservedRunningTime="2026-09-16 18:42:30.346213784 +0000 UTC m=+134.065310858"
Sep 16 18:42:31.308188 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.308152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc7ds" event={"ID":"7633ccec-856e-413c-8d65-3aaeb1201b6a","Type":"ContainerStarted","Data":"d0230c508fa57903577939c5bb93ef371947ec5287c494d8d2f16e552dc9b6bc"}
Sep 16 18:42:31.308790 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.308218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc7ds" event={"ID":"7633ccec-856e-413c-8d65-3aaeb1201b6a","Type":"ContainerStarted","Data":"036630193a94ac826519edc39d1c229af56f329d25f8fd18083de57743a30428"}
Sep 16 18:42:31.311875 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.311845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj85s" event={"ID":"f2edca00-829c-4dc5-8731-99bce57a60d3","Type":"ContainerStarted","Data":"0c9180b714a4f2a961e7d2ab0fb848fea1ab09e9b0c896aaa4a6aceb1088dfe0"}
Sep 16 18:42:31.312003 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.311903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pj85s" event={"ID":"f2edca00-829c-4dc5-8731-99bce57a60d3","Type":"ContainerStarted","Data":"05a9ff03ad4c24baf39c830d914636aa7c51c4fc793ae1f9dc266e384fe4bbc4"}
Sep 16 18:42:31.327612 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.327561    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bc7ds" podStartSLOduration=132.536038394 podStartE2EDuration="2m15.327542458s" totalImagesPullingTime="2.791504064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:31.32478485 +0000 UTC m=+135.043881928" watchObservedRunningTime="2026-09-16 18:42:31.327542458 +0000 UTC m=+135.046639535"
Sep 16 18:42:31.348248 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:31.347776    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pj85s" podStartSLOduration=6.303732717 podStartE2EDuration="10.347754626s" totalImagesPullingTime="3.90289738s" totalInitContainerRuntime="141.124529ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:31.346299076 +0000 UTC m=+135.065396154" watchObservedRunningTime="2026-09-16 18:42:31.347754626 +0000 UTC m=+135.066851704"
Sep 16 18:42:32.321488 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:32.321438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"aa5623c359f3ead889ffb5aad02c6868d87b7bd2cd19185594293567537b8135"}
Sep 16 18:42:32.321811 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:32.321520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"e4686b38936b98fbef421c6ddadd7babac6d8669d652e6c692787fd439a2fa88"}
Sep 16 18:42:33.133295 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.133227    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" podUID="7ec644e7-103c-4d0b-9403-50219832b385" containerName="registry" containerID="cri-o://b410be50ad71ee08c269e941d6359c9c1cf82ed72af8626b62c3942be3cc46a5" gracePeriod=30
Sep 16 18:42:33.332638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.330097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" event={"ID":"b328323d-de90-4291-bb87-8011bba6dbaf","Type":"ContainerStarted","Data":"9e9e0ffd31e14a8e7c187e65469e371fc73a21c61f67b20f5f9230238a1f8754"}
Sep 16 18:42:33.332638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.330313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:33.332638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.332089    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ec644e7-103c-4d0b-9403-50219832b385" containerID="b410be50ad71ee08c269e941d6359c9c1cf82ed72af8626b62c3942be3cc46a5" exitCode=0
Sep 16 18:42:33.332638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.332167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" event={"ID":"7ec644e7-103c-4d0b-9403-50219832b385","Type":"ContainerDied","Data":"b410be50ad71ee08c269e941d6359c9c1cf82ed72af8626b62c3942be3cc46a5"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"a97278396969be0a5315bcd813516dd67ef23089fc8f7872df156a73e8d2af1e"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"0247fb161ad2d26f9dcc361cbbfc3ddfb587e9db4d78a9f9b5e777d38e9d6854"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"2cce97d12150d75fcdd929ce87c6abbdfd556b756ef0bb21972bf1f1ec1154c0"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"570ffe1dd3e2ae5c1770736a06cd7399ab7ca7f9dca24e620a60f3cf70948d90"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"b3b0aa1216e28a51815bcd93175d9de13f4387b63dfe91c361f47f10cf087c01"}
Sep 16 18:42:33.337559 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.337506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"826d7ec7-6587-43c4-9279-4d97ccac5b85","Type":"ContainerStarted","Data":"a7362dc9b681604fe8c257518db5968a0df0d7fafbea8c42389fd5829ce03edd"}
Sep 16 18:42:33.363045 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.362564    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9" podStartSLOduration=3.728887657 podStartE2EDuration="9.362542815s" totalImagesPullingTime="5.633655158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:33.357847338 +0000 UTC m=+137.076944418" watchObservedRunningTime="2026-09-16 18:42:33.362542815 +0000 UTC m=+137.081639898"
Sep 16 18:42:33.387021 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.386962    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.7652790849999995 podStartE2EDuration="11.386938554s" totalImagesPullingTime="5.595916547s" totalInitContainerRuntime="25.742922ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:33.384614728 +0000 UTC m=+137.103711829" watchObservedRunningTime="2026-09-16 18:42:33.386938554 +0000 UTC m=+137.106035632"
Sep 16 18:42:33.441033 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.441003    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:42:33.524581 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524552    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qtwt\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.524954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.524831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration\") pod \"7ec644e7-103c-4d0b-9403-50219832b385\" (UID: \"7ec644e7-103c-4d0b-9403-50219832b385\") "
Sep 16 18:42:33.528255 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.525101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:33.528255 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.525280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:33.528255 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.528101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:33.528494 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.528297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:33.528494 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.528307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:33.528494 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.528320    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:33.529267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.529245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt" (OuterVolumeSpecName: "kube-api-access-6qtwt") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "kube-api-access-6qtwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:33.535485 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.535427    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7ec644e7-103c-4d0b-9403-50219832b385" (UID: "7ec644e7-103c-4d0b-9403-50219832b385"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626431    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-registry-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626476    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-trusted-ca\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626491    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-installation-pull-secrets\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626503    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ec644e7-103c-4d0b-9403-50219832b385-ca-trust-extracted\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626513    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ec644e7-103c-4d0b-9403-50219832b385-image-registry-private-configuration\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626523    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qtwt\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-kube-api-access-6qtwt\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626828 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626531    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ec644e7-103c-4d0b-9403-50219832b385-bound-sa-token\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:33.626828 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:33.626539    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ec644e7-103c-4d0b-9403-50219832b385-registry-certificates\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:42:34.343508 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.343447    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85"
Sep 16 18:42:34.343508 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.343500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffd4b88cd-pnj85" event={"ID":"7ec644e7-103c-4d0b-9403-50219832b385","Type":"ContainerDied","Data":"f0bce2e20b08d233d63cae24b7b3f1f5701aebc246314eca5dfb5c37f090078a"}
Sep 16 18:42:34.344029 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.343561    2579 scope.go:117] "RemoveContainer" containerID="b410be50ad71ee08c269e941d6359c9c1cf82ed72af8626b62c3942be3cc46a5"
Sep 16 18:42:34.352326 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.352304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ffcfc849c-ghbc9"
Sep 16 18:42:34.391979 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.391950    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:42:34.396334 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.396307    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ffd4b88cd-pnj85"]
Sep 16 18:42:34.749378 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:42:34.749300    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec644e7-103c-4d0b-9403-50219832b385" path="/var/lib/kubelet/pods/7ec644e7-103c-4d0b-9403-50219832b385/volumes"
Sep 16 18:43:06.466135 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:06.466038    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0db4211-6f85-4e65-9cb7-bd14ae7f782f" containerID="b0bf1b919678348cef361b180e8b1b1d2d0c47cdfa45871e9e683a66fa474361" exitCode=0
Sep 16 18:43:06.466135 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:06.466085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm" event={"ID":"d0db4211-6f85-4e65-9cb7-bd14ae7f782f","Type":"ContainerDied","Data":"b0bf1b919678348cef361b180e8b1b1d2d0c47cdfa45871e9e683a66fa474361"}
Sep 16 18:43:06.466652 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:06.466374    2579 scope.go:117] "RemoveContainer" containerID="b0bf1b919678348cef361b180e8b1b1d2d0c47cdfa45871e9e683a66fa474361"
Sep 16 18:43:07.471577 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:07.471544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lqzxm" event={"ID":"d0db4211-6f85-4e65-9cb7-bd14ae7f782f","Type":"ContainerStarted","Data":"ba745421301c6067d7deba28c19319a63e796d52d8e46f3d516ad46de3475e9e"}
Sep 16 18:43:11.493809 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:11.493775    2579 generic.go:356] "Generic (PLEG): container finished" podID="ff45e963-083a-410b-a861-7f5379c427ac" containerID="1c933ef7d812b853cca8549982c556a189f57d4073963241bb53ea3a4a79c8e7" exitCode=0
Sep 16 18:43:11.494223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:11.493859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qsnj6" event={"ID":"ff45e963-083a-410b-a861-7f5379c427ac","Type":"ContainerDied","Data":"1c933ef7d812b853cca8549982c556a189f57d4073963241bb53ea3a4a79c8e7"}
Sep 16 18:43:11.494267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:11.494225    2579 scope.go:117] "RemoveContainer" containerID="1c933ef7d812b853cca8549982c556a189f57d4073963241bb53ea3a4a79c8e7"
Sep 16 18:43:12.499893 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:43:12.499858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qsnj6" event={"ID":"ff45e963-083a-410b-a861-7f5379c427ac","Type":"ContainerStarted","Data":"f48a89119b335b8ad185155dc8f94f8de2515839f2233113661ef5bf3086037c"}
Sep 16 18:45:16.662207 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:16.662179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:45:16.664334 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:16.664299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:45:16.669780 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:16.669757    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:39.077620 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.077589    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9dkcp"]
Sep 16 18:45:39.079046 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.077909    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec644e7-103c-4d0b-9403-50219832b385" containerName="registry"
Sep 16 18:45:39.079046 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.077921    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec644e7-103c-4d0b-9403-50219832b385" containerName="registry"
Sep 16 18:45:39.079046 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.077989    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec644e7-103c-4d0b-9403-50219832b385" containerName="registry"
Sep 16 18:45:39.079602 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.079587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.082348 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.082317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:45:39.082348 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.082334    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:45:39.082348 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.082341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-d2kf4\""
Sep 16 18:45:39.082556 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.082324    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:45:39.083394 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.083374    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:45:39.083394 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.083387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:45:39.093542 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.093514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9dkcp"]
Sep 16 18:45:39.098200 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.098172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-cabundle0\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.098319 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.098234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xmk\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-kube-api-access-j6xmk\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.098319 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.098291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.198801 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.198765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-cabundle0\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.198815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xmk\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-kube-api-access-j6xmk\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.198843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:45:39.198947    2579 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:45:39.198961    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:45:39.198968    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:45:39.198975 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:45:39.198979    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-9dkcp: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:45:39.199283 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:45:39.199030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates podName:c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.699016777 +0000 UTC m=+323.418113832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates") pod "keda-operator-579c8958f7-9dkcp" (UID: "c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:45:39.199500 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.199479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-cabundle0\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.207869 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.207845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xmk\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-kube-api-access-j6xmk\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.702632 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.702596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.705039 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.705021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a-certificates\") pod \"keda-operator-579c8958f7-9dkcp\" (UID: \"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a\") " pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:39.989404 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:39.989314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:40.259691 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:40.259543    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9dkcp"]
Sep 16 18:45:40.262066 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:45:40.262039    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9878b0f_53d5_49f5_8b1b_b3acd16fbb5a.slice/crio-def94582ac1f65a1c13a745ac75752da5ac0bb12e550f35dd49be3cec56cd988 WatchSource:0}: Error finding container def94582ac1f65a1c13a745ac75752da5ac0bb12e550f35dd49be3cec56cd988: Status 404 returned error can't find the container with id def94582ac1f65a1c13a745ac75752da5ac0bb12e550f35dd49be3cec56cd988
Sep 16 18:45:40.263297 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:40.263283    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:41.072191 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:41.072156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-9dkcp" event={"ID":"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a","Type":"ContainerStarted","Data":"def94582ac1f65a1c13a745ac75752da5ac0bb12e550f35dd49be3cec56cd988"}
Sep 16 18:45:44.086014 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:44.085976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-9dkcp" event={"ID":"c9878b0f-53d5-49f5-8b1b-b3acd16fbb5a","Type":"ContainerStarted","Data":"7568ac97150d3a93f7076827088b32ecad6df56e13dda31242c1462aeaba11e7"}
Sep 16 18:45:44.086379 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:44.086116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:45:44.107220 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:45:44.107167    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-9dkcp" podStartSLOduration=1.454417871 podStartE2EDuration="5.107152674s" totalImagesPullingTime="3.652734803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:44.105292666 +0000 UTC m=+327.824389742" watchObservedRunningTime="2026-09-16 18:45:44.107152674 +0000 UTC m=+327.826249751"
Sep 16 18:46:05.092932 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:05.092899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-9dkcp"
Sep 16 18:46:56.458698 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.458667    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:46:56.461961 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.461944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:56.464429 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.464409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:46:56.464544 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.464436    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:46:56.465383 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.465357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:46:56.465515 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.465492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8rn4t\""
Sep 16 18:46:56.470675 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.470654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:46:56.520056 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.520019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:56.520212 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.520111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cgs\" (UniqueName: \"kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:56.620812 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.620779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:56.621000 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.620866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cgs\" (UniqueName: \"kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:56.621000 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:46:56.620946    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:46:56.621099 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:46:56.621013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert podName:b2ce9514-30f1-4543-a6e1-3681c68cbde3 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:57.120995562 +0000 UTC m=+400.840092635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert") pod "kserve-controller-manager-557b59fdc4-krd9t" (UID: "b2ce9514-30f1-4543-a6e1-3681c68cbde3") : secret "kserve-webhook-server-cert" not found
Sep 16 18:46:56.630417 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:56.630393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cgs\" (UniqueName: \"kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:57.124234 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:57.124197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:57.126716 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:57.126693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") pod \"kserve-controller-manager-557b59fdc4-krd9t\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") " pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:57.372596 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:57.372569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:46:57.643986 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:57.643909    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:46:57.647598 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:46:57.647570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ce9514_30f1_4543_a6e1_3681c68cbde3.slice/crio-6f40e9d02f35b143021da7401a0cf757902fcf2578301f41725b8edaa3adcd1a WatchSource:0}: Error finding container 6f40e9d02f35b143021da7401a0cf757902fcf2578301f41725b8edaa3adcd1a: Status 404 returned error can't find the container with id 6f40e9d02f35b143021da7401a0cf757902fcf2578301f41725b8edaa3adcd1a
Sep 16 18:46:58.370142 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:46:58.370097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" event={"ID":"b2ce9514-30f1-4543-a6e1-3681c68cbde3","Type":"ContainerStarted","Data":"6f40e9d02f35b143021da7401a0cf757902fcf2578301f41725b8edaa3adcd1a"}
Sep 16 18:47:01.385033 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:01.385000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" event={"ID":"b2ce9514-30f1-4543-a6e1-3681c68cbde3","Type":"ContainerStarted","Data":"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"}
Sep 16 18:47:01.385477 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:01.385111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:47:01.402208 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:01.402162    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" podStartSLOduration=2.147250193 podStartE2EDuration="5.402148912s" totalImagesPullingTime="3.254898719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.400672301 +0000 UTC m=+405.119769390" watchObservedRunningTime="2026-09-16 18:47:01.402148912 +0000 UTC m=+405.121245989"
Sep 16 18:47:32.394684 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:32.394656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:47:40.022676 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.022594    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:47:40.023097 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.022789    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" podUID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" containerName="manager" containerID="cri-o://bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2" gracePeriod=10
Sep 16 18:47:40.050678 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.050649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-rf7m9"]
Sep 16 18:47:40.053041 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.053021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.065694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.065673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-rf7m9"]
Sep 16 18:47:40.184419 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.184387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-cert\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.184419 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.184424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqx2\" (UniqueName: \"kubernetes.io/projected/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-kube-api-access-kvqx2\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.258655 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.258634    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:47:40.285903 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.285822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqx2\" (UniqueName: \"kubernetes.io/projected/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-kube-api-access-kvqx2\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.286036 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.286002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-cert\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.288619 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.288593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-cert\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.295099 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.295075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqx2\" (UniqueName: \"kubernetes.io/projected/3b1b4d00-04a5-4d8a-b590-095ad3ea74b6-kube-api-access-kvqx2\") pod \"kserve-controller-manager-557b59fdc4-rf7m9\" (UID: \"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6\") " pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.387008 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.386970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9cgs\" (UniqueName: \"kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs\") pod \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") "
Sep 16 18:47:40.387160 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.387027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") pod \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\" (UID: \"b2ce9514-30f1-4543-a6e1-3681c68cbde3\") "
Sep 16 18:47:40.389319 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.389281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert" (OuterVolumeSpecName: "cert") pod "b2ce9514-30f1-4543-a6e1-3681c68cbde3" (UID: "b2ce9514-30f1-4543-a6e1-3681c68cbde3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:40.389424 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.389329    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs" (OuterVolumeSpecName: "kube-api-access-d9cgs") pod "b2ce9514-30f1-4543-a6e1-3681c68cbde3" (UID: "b2ce9514-30f1-4543-a6e1-3681c68cbde3"). InnerVolumeSpecName "kube-api-access-d9cgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:40.401527 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.401505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:40.487592 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.487565    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9cgs\" (UniqueName: \"kubernetes.io/projected/b2ce9514-30f1-4543-a6e1-3681c68cbde3-kube-api-access-d9cgs\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:47:40.487760 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.487590    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ce9514-30f1-4543-a6e1-3681c68cbde3-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:47:40.543618 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.543538    2579 generic.go:356] "Generic (PLEG): container finished" podID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" containerID="bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2" exitCode=0
Sep 16 18:47:40.543618 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.543570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" event={"ID":"b2ce9514-30f1-4543-a6e1-3681c68cbde3","Type":"ContainerDied","Data":"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"}
Sep 16 18:47:40.543618 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.543585    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t"
Sep 16 18:47:40.543858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.543655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-krd9t" event={"ID":"b2ce9514-30f1-4543-a6e1-3681c68cbde3","Type":"ContainerDied","Data":"6f40e9d02f35b143021da7401a0cf757902fcf2578301f41725b8edaa3adcd1a"}
Sep 16 18:47:40.543858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.543683    2579 scope.go:117] "RemoveContainer" containerID="bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"
Sep 16 18:47:40.552931 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.552911    2579 scope.go:117] "RemoveContainer" containerID="bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"
Sep 16 18:47:40.553198 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:47:40.553181    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2\": container with ID starting with bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2 not found: ID does not exist" containerID="bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"
Sep 16 18:47:40.553264 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.553205    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2"} err="failed to get container status \"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2\": rpc error: code = NotFound desc = could not find container \"bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2\": container with ID starting with bae42f356496097714bea03776b77a2a472477c2a6531850452084970b0782c2 not found: ID does not exist"
Sep 16 18:47:40.572494 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.572473    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:47:40.591942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.591915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-krd9t"]
Sep 16 18:47:40.668199 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.668179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-rf7m9"]
Sep 16 18:47:40.670820 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:47:40.670797    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1b4d00_04a5_4d8a_b590_095ad3ea74b6.slice/crio-d0efa451f2aeb81e6f016cd47382786520ef0b57c1ee5e190d958eb45d938053 WatchSource:0}: Error finding container d0efa451f2aeb81e6f016cd47382786520ef0b57c1ee5e190d958eb45d938053: Status 404 returned error can't find the container with id d0efa451f2aeb81e6f016cd47382786520ef0b57c1ee5e190d958eb45d938053
Sep 16 18:47:40.748901 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:40.748871    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" path="/var/lib/kubelet/pods/b2ce9514-30f1-4543-a6e1-3681c68cbde3/volumes"
Sep 16 18:47:41.549282 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:41.549249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9" event={"ID":"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6","Type":"ContainerStarted","Data":"d19309ef8685d86d0cd97ff337b1026c027214b041c37afa19d8fe9cbc6faa2a"}
Sep 16 18:47:41.549763 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:41.549299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9" event={"ID":"3b1b4d00-04a5-4d8a-b590-095ad3ea74b6","Type":"ContainerStarted","Data":"d0efa451f2aeb81e6f016cd47382786520ef0b57c1ee5e190d958eb45d938053"}
Sep 16 18:47:41.549763 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:41.549341    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:47:41.567883 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:47:41.567840    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9" podStartSLOduration=1.27786244 podStartE2EDuration="1.567826666s" totalImagesPullingTime="289.964226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.566805358 +0000 UTC m=+445.285902438" watchObservedRunningTime="2026-09-16 18:47:41.567826666 +0000 UTC m=+445.286923743"
Sep 16 18:48:12.559749 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:12.559719    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-557b59fdc4-rf7m9"
Sep 16 18:48:13.921629 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.921599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-nw8vm"]
Sep 16 18:48:13.921993 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.921918    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" containerName="manager"
Sep 16 18:48:13.921993 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.921928    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" containerName="manager"
Sep 16 18:48:13.921993 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.921981    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ce9514-30f1-4543-a6e1-3681c68cbde3" containerName="manager"
Sep 16 18:48:13.924717 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.924696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:13.928135 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.928119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:48:13.928571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.928549    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xwc92\""
Sep 16 18:48:13.937409 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.937387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-nw8vm"]
Sep 16 18:48:13.967719 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.967690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:13.967829 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:13.967748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lb7\" (UniqueName: \"kubernetes.io/projected/3e474aa1-a3d3-4a22-96a3-4442ab26d624-kube-api-access-d7lb7\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.068748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.068716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.069017 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.068763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lb7\" (UniqueName: \"kubernetes.io/projected/3e474aa1-a3d3-4a22-96a3-4442ab26d624-kube-api-access-d7lb7\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.069017 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:48:14.068871    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:48:14.069017 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:48:14.068945    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs podName:3e474aa1-a3d3-4a22-96a3-4442ab26d624 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:14.568929338 +0000 UTC m=+478.288026392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs") pod "model-serving-api-7b954ffc95-nw8vm" (UID: "3e474aa1-a3d3-4a22-96a3-4442ab26d624") : secret "model-serving-api-tls" not found
Sep 16 18:48:14.078476 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.078432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lb7\" (UniqueName: \"kubernetes.io/projected/3e474aa1-a3d3-4a22-96a3-4442ab26d624-kube-api-access-d7lb7\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.572465 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.572430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.574940 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.574906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e474aa1-a3d3-4a22-96a3-4442ab26d624-tls-certs\") pod \"model-serving-api-7b954ffc95-nw8vm\" (UID: \"3e474aa1-a3d3-4a22-96a3-4442ab26d624\") " pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:14.834720 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:14.834626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:15.106983 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:15.106814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-nw8vm"]
Sep 16 18:48:15.109516 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:48:15.109491    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e474aa1_a3d3_4a22_96a3_4442ab26d624.slice/crio-31f554472222a154e2047dd6ac3857e2ddab697e4325b86cb60a52bdcb8f0ce4 WatchSource:0}: Error finding container 31f554472222a154e2047dd6ac3857e2ddab697e4325b86cb60a52bdcb8f0ce4: Status 404 returned error can't find the container with id 31f554472222a154e2047dd6ac3857e2ddab697e4325b86cb60a52bdcb8f0ce4
Sep 16 18:48:15.693398 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:15.693354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-nw8vm" event={"ID":"3e474aa1-a3d3-4a22-96a3-4442ab26d624","Type":"ContainerStarted","Data":"31f554472222a154e2047dd6ac3857e2ddab697e4325b86cb60a52bdcb8f0ce4"}
Sep 16 18:48:17.703843 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:17.703809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-nw8vm" event={"ID":"3e474aa1-a3d3-4a22-96a3-4442ab26d624","Type":"ContainerStarted","Data":"f923ba06f7884c3ae5f22193a27ff384029aa073786a4347871df6bffb910eb5"}
Sep 16 18:48:17.704341 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:17.703943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:17.724878 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:17.724835    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-nw8vm" podStartSLOduration=2.621916345 podStartE2EDuration="4.724821448s" totalImagesPullingTime="2.102905103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.723381748 +0000 UTC m=+481.442478825" watchObservedRunningTime="2026-09-16 18:48:17.724821448 +0000 UTC m=+481.443918524"
Sep 16 18:48:28.711792 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:28.711757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-nw8vm"
Sep 16 18:48:30.652298 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.652260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-27t59"]
Sep 16 18:48:30.658793 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.658768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-27t59"
Sep 16 18:48:30.661975 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.661949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:48:30.662102 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.661994    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:48:30.662102 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.662016    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g47rj\""
Sep 16 18:48:30.669748 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.669725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-27t59"]
Sep 16 18:48:30.708192 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.708171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.708296 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.708207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7zv\" (UniqueName: \"kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.809193 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.809157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.809349 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.809207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7zv\" (UniqueName: \"kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.809616 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.809593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.819313 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.819281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7zv\" (UniqueName: \"kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv\") pod \"s3-init-27t59\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") " pod="kserve/s3-init-27t59"
Sep 16 18:48:30.975905 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:30.975826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-27t59"
Sep 16 18:48:31.244833 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:31.244758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-27t59"]
Sep 16 18:48:31.249047 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:48:31.249021    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaf286e2_8bc4_4e6b_bac5_faac0d3bdbed.slice/crio-e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2 WatchSource:0}: Error finding container e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2: Status 404 returned error can't find the container with id e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2
Sep 16 18:48:31.759362 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:31.759331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerStarted","Data":"e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2"}
Sep 16 18:48:35.777298 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:35.777210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerStarted","Data":"77ae62ece70c47a0ff1a5991af410b88103f67c781a7a318e37e995e2fc0b732"}
Sep 16 18:48:37.787942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:37.787849    2579 generic.go:356] "Generic (PLEG): container finished" podID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerID="77ae62ece70c47a0ff1a5991af410b88103f67c781a7a318e37e995e2fc0b732" exitCode=0
Sep 16 18:48:37.787942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:37.787906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerDied","Data":"77ae62ece70c47a0ff1a5991af410b88103f67c781a7a318e37e995e2fc0b732"}
Sep 16 18:48:42.814648 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:42.814614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerStarted","Data":"93d3b514ce5327a6116c3ff1ab5379d9056973b206241e871ef982021d2a7ae7"}
Sep 16 18:48:42.830212 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:48:42.830159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-27t59" podStartSLOduration=2.227240716 podStartE2EDuration="12.83013873s" totalImagesPullingTime="8.755377021s" totalInitContainerRuntime="1.847520993s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:42.830044105 +0000 UTC m=+506.549141176" watchObservedRunningTime="2026-09-16 18:48:42.83013873 +0000 UTC m=+506.549235814"
Sep 16 18:49:48.084239 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:48.084205    2579 generic.go:356] "Generic (PLEG): container finished" podID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerID="93d3b514ce5327a6116c3ff1ab5379d9056973b206241e871ef982021d2a7ae7" exitCode=0
Sep 16 18:49:48.084644 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:48.084274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerDied","Data":"93d3b514ce5327a6116c3ff1ab5379d9056973b206241e871ef982021d2a7ae7"}
Sep 16 18:49:49.212616 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.212588    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-27t59"
Sep 16 18:49:49.357757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.357666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7zv\" (UniqueName: \"kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv\") pod \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") "
Sep 16 18:49:49.357757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.357721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache\") pod \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\" (UID: \"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed\") "
Sep 16 18:49:49.359872 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.359846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv" (OuterVolumeSpecName: "kube-api-access-6r7zv") pod "aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" (UID: "aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed"). InnerVolumeSpecName "kube-api-access-6r7zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:49.379593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.379546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache" (OuterVolumeSpecName: "model-cache") pod "aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" (UID: "aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:49.458381 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.458354    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7zv\" (UniqueName: \"kubernetes.io/projected/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-kube-api-access-6r7zv\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:49:49.458381 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:49.458376    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed-model-cache\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:49:50.093871 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:50.093838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-27t59" event={"ID":"aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed","Type":"ContainerDied","Data":"e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2"}
Sep 16 18:49:50.094050 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:50.093883    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71520a0ce98e113f02d77e865b86f9a7a8195bcf920c47874a8aace292bbab2"
Sep 16 18:49:50.094050 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:50.093844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-27t59"
Sep 16 18:49:56.308974 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.308943    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"]
Sep 16 18:49:56.309373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.309264    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerName="s3-init"
Sep 16 18:49:56.309373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.309275    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerName="s3-init"
Sep 16 18:49:56.309373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.309287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerName="download-hf-model"
Sep 16 18:49:56.309373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.309294    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerName="download-hf-model"
Sep 16 18:49:56.309373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.309342    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed" containerName="s3-init"
Sep 16 18:49:56.311389 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.311375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.314137 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.314119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g47rj\""
Sep 16 18:49:56.315114 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.315092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:49:56.315225 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.315128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 16 18:49:56.320105 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.320086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"]
Sep 16 18:49:56.416997 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.416954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.417181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.417018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klb49\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-kube-api-access-klb49\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.417181 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.417063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aff80783-128f-4457-9323-868d469118b0-data\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.517502 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.517447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aff80783-128f-4457-9323-868d469118b0-data\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.517639 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.517545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.517639 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.517603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klb49\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-kube-api-access-klb49\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.517892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.517870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aff80783-128f-4457-9323-868d469118b0-data\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.520025 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.520005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.526659 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.526635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klb49\" (UniqueName: \"kubernetes.io/projected/aff80783-128f-4457-9323-868d469118b0-kube-api-access-klb49\") pod \"seaweedfs-tls-custom-8b44d767d-lmzgd\" (UID: \"aff80783-128f-4457-9323-868d469118b0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.621064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.620985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"
Sep 16 18:49:56.889891 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:56.889835    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-lmzgd"]
Sep 16 18:49:56.892564 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:49:56.892533    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff80783_128f_4457_9323_868d469118b0.slice/crio-bd08a9e70653b948ffc73f96f94621496c9599d3a62a9fba44d4e612ff4de85a WatchSource:0}: Error finding container bd08a9e70653b948ffc73f96f94621496c9599d3a62a9fba44d4e612ff4de85a: Status 404 returned error can't find the container with id bd08a9e70653b948ffc73f96f94621496c9599d3a62a9fba44d4e612ff4de85a
Sep 16 18:49:57.124270 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:49:57.124242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd" event={"ID":"aff80783-128f-4457-9323-868d469118b0","Type":"ContainerStarted","Data":"bd08a9e70653b948ffc73f96f94621496c9599d3a62a9fba44d4e612ff4de85a"}
Sep 16 18:50:00.139565 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.139534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd" event={"ID":"aff80783-128f-4457-9323-868d469118b0","Type":"ContainerStarted","Data":"0692cb9648ee775c04f8aa56fa64121e1e5b42934d38eb1f36ce69bd730f658a"}
Sep 16 18:50:00.157722 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.157680    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-lmzgd" podStartSLOduration=1.471333789 podStartE2EDuration="4.157666912s" totalImagesPullingTime="2.686333123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:00.156887994 +0000 UTC m=+583.875985071" watchObservedRunningTime="2026-09-16 18:50:00.157666912 +0000 UTC m=+583.876763989"
Sep 16 18:50:00.436616 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.436524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-9blhl"]
Sep 16 18:50:00.439626 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.439602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.442084 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.442064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:50:00.447167 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.447145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-9blhl"]
Sep 16 18:50:00.454554 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.454533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.454640 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.454573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kdk\" (UniqueName: \"kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.555072 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.555037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.555253 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.555095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kdk\" (UniqueName: \"kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.555447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.555426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.563413 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.563391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kdk\" (UniqueName: \"kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk\") pod \"s3-tls-init-custom-9blhl\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") " pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:00.762080 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:00.762003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:01.031562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:01.031540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-9blhl"]
Sep 16 18:50:01.033809 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:50:01.033783    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4e26eff_2d34_4c75_b192_3666e2b9c1a3.slice/crio-59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45 WatchSource:0}: Error finding container 59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45: Status 404 returned error can't find the container with id 59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45
Sep 16 18:50:01.145938 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:01.145901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerStarted","Data":"67d2061a5efbf758f524cd9eff58007faca8c3e1ea984ac160d309d4600fdc6a"}
Sep 16 18:50:01.146322 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:01.145945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerStarted","Data":"59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45"}
Sep 16 18:50:03.164838 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:03.164800    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerID="67d2061a5efbf758f524cd9eff58007faca8c3e1ea984ac160d309d4600fdc6a" exitCode=0
Sep 16 18:50:03.165274 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:03.164872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerDied","Data":"67d2061a5efbf758f524cd9eff58007faca8c3e1ea984ac160d309d4600fdc6a"}
Sep 16 18:50:04.170591 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:04.170559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerStarted","Data":"695c368065e7fd9a5e700226bea7afe5b569349fd39427c36c484a200e139213"}
Sep 16 18:50:04.187074 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:04.187026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-9blhl" podStartSLOduration=2.880687918 podStartE2EDuration="4.187008334s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.306320416s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:04.186076525 +0000 UTC m=+587.905173627" watchObservedRunningTime="2026-09-16 18:50:04.187008334 +0000 UTC m=+587.906105593"
Sep 16 18:50:10.201470 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:10.201419    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerID="695c368065e7fd9a5e700226bea7afe5b569349fd39427c36c484a200e139213" exitCode=0
Sep 16 18:50:10.201929 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:10.201498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerDied","Data":"695c368065e7fd9a5e700226bea7afe5b569349fd39427c36c484a200e139213"}
Sep 16 18:50:11.346727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.346702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:11.432743 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.432708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4kdk\" (UniqueName: \"kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk\") pod \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") "
Sep 16 18:50:11.432888 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.432783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache\") pod \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\" (UID: \"f4e26eff-2d34-4c75-b192-3666e2b9c1a3\") "
Sep 16 18:50:11.434895 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.434867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk" (OuterVolumeSpecName: "kube-api-access-x4kdk") pod "f4e26eff-2d34-4c75-b192-3666e2b9c1a3" (UID: "f4e26eff-2d34-4c75-b192-3666e2b9c1a3"). InnerVolumeSpecName "kube-api-access-x4kdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:11.453837 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.453812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4e26eff-2d34-4c75-b192-3666e2b9c1a3" (UID: "f4e26eff-2d34-4c75-b192-3666e2b9c1a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:11.534106 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.534018    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-model-cache\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:11.534106 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:11.534046    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kdk\" (UniqueName: \"kubernetes.io/projected/f4e26eff-2d34-4c75-b192-3666e2b9c1a3-kube-api-access-x4kdk\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:12.211521 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:12.211493    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9blhl"
Sep 16 18:50:12.211683 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:12.211493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9blhl" event={"ID":"f4e26eff-2d34-4c75-b192-3666e2b9c1a3","Type":"ContainerDied","Data":"59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45"}
Sep 16 18:50:12.211683 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:12.211599    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59622fa94db3804a85e24838b507e39c82e97ab24385cbda47503c85faf98e45"
Sep 16 18:50:14.889843 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.889808    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-smn88"]
Sep 16 18:50:14.890223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.890136    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerName="s3-tls-init-custom"
Sep 16 18:50:14.890223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.890147    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerName="s3-tls-init-custom"
Sep 16 18:50:14.890223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.890158    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerName="download-hf-model"
Sep 16 18:50:14.890223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.890163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerName="download-hf-model"
Sep 16 18:50:14.890223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.890214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4e26eff-2d34-4c75-b192-3666e2b9c1a3" containerName="s3-tls-init-custom"
Sep 16 18:50:14.892979 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.892956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:14.895589 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.895566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:50:14.895589 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.895570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:50:14.901673 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.901649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-smn88"]
Sep 16 18:50:14.963433 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.963402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:14.963433 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:14.963437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xh7r\" (UniqueName: \"kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.064021 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.063966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.064021 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.064029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xh7r\" (UniqueName: \"kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.064372 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.064337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.073595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.073568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xh7r\" (UniqueName: \"kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r\") pod \"s3-tls-init-serving-smn88\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") " pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.212358 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.212277    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:15.481420 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:15.481396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-smn88"]
Sep 16 18:50:15.484065 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:50:15.484033    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a07856c_fd6a_400c_b6fc_abce7655cd18.slice/crio-739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced WatchSource:0}: Error finding container 739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced: Status 404 returned error can't find the container with id 739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced
Sep 16 18:50:16.229390 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:16.229358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerStarted","Data":"2fca2b0c7ec20861459184331c8a7b4257b18be50297580e39609413141e7f4c"}
Sep 16 18:50:16.229390 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:16.229396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerStarted","Data":"739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced"}
Sep 16 18:50:16.705615 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:16.705585    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:50:16.705771 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:16.705650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:50:18.239235 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:18.239199    2579 generic.go:356] "Generic (PLEG): container finished" podID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerID="2fca2b0c7ec20861459184331c8a7b4257b18be50297580e39609413141e7f4c" exitCode=0
Sep 16 18:50:18.239683 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:18.239252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerDied","Data":"2fca2b0c7ec20861459184331c8a7b4257b18be50297580e39609413141e7f4c"}
Sep 16 18:50:18.248763 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:18.248746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:50:19.245384 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:19.245347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerStarted","Data":"7e5cf37a41e675c5255c697bcf0d8b8dbb9ba83b77623a49bf4a0c3096a9ed3e"}
Sep 16 18:50:19.262851 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:19.262803    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-smn88" podStartSLOduration=3.460072648 podStartE2EDuration="5.26278663s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.802713982s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:19.261103503 +0000 UTC m=+602.980200591" watchObservedRunningTime="2026-09-16 18:50:19.26278663 +0000 UTC m=+602.981883706"
Sep 16 18:50:25.273535 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:25.273503    2579 generic.go:356] "Generic (PLEG): container finished" podID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerID="7e5cf37a41e675c5255c697bcf0d8b8dbb9ba83b77623a49bf4a0c3096a9ed3e" exitCode=0
Sep 16 18:50:25.273992 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:25.273582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerDied","Data":"7e5cf37a41e675c5255c697bcf0d8b8dbb9ba83b77623a49bf4a0c3096a9ed3e"}
Sep 16 18:50:26.412128 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.412105    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:26.461587 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.461562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xh7r\" (UniqueName: \"kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r\") pod \"7a07856c-fd6a-400c-b6fc-abce7655cd18\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") "
Sep 16 18:50:26.461713 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.461630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache\") pod \"7a07856c-fd6a-400c-b6fc-abce7655cd18\" (UID: \"7a07856c-fd6a-400c-b6fc-abce7655cd18\") "
Sep 16 18:50:26.463750 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.463727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r" (OuterVolumeSpecName: "kube-api-access-2xh7r") pod "7a07856c-fd6a-400c-b6fc-abce7655cd18" (UID: "7a07856c-fd6a-400c-b6fc-abce7655cd18"). InnerVolumeSpecName "kube-api-access-2xh7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:26.487335 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.487304    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a07856c-fd6a-400c-b6fc-abce7655cd18" (UID: "7a07856c-fd6a-400c-b6fc-abce7655cd18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:26.562817 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.562792    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a07856c-fd6a-400c-b6fc-abce7655cd18-model-cache\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:26.562817 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:26.562817    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xh7r\" (UniqueName: \"kubernetes.io/projected/7a07856c-fd6a-400c-b6fc-abce7655cd18-kube-api-access-2xh7r\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:27.283867 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:27.283841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-smn88"
Sep 16 18:50:27.284037 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:27.283836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-smn88" event={"ID":"7a07856c-fd6a-400c-b6fc-abce7655cd18","Type":"ContainerDied","Data":"739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced"}
Sep 16 18:50:27.284037 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:27.283959    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739a746195035faf9ae186d2c6acc711ae97e9509b44ee9a10cd7820a8167ced"
Sep 16 18:50:39.373287 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373254    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:50:39.375593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373612    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerName="download-hf-model"
Sep 16 18:50:39.375593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerName="download-hf-model"
Sep 16 18:50:39.375593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373639    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerName="s3-tls-init-serving"
Sep 16 18:50:39.375593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerName="s3-tls-init-serving"
Sep 16 18:50:39.375593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.373701    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a07856c-fd6a-400c-b6fc-abce7655cd18" containerName="s3-tls-init-serving"
Sep 16 18:50:39.376764 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.376748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.379527 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:50:39.379663 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379559    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 18:50:39.379663 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 18:50:39.379663 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:50:39.379837 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:50:39.379983 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.379966    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:50:39.380501 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.380484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 18:50:39.388537 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.388513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:50:39.468220 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.468190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.468220 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.468220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvps\" (UniqueName: \"kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.468402 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.468248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.468402 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.468340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.468402 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.468363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.477961 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.477937    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:39.481221 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.481202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.483902 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.483881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-fce992\""
Sep 16 18:50:39.483902 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.483892    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fce992-kube-rbac-proxy-sar-config\""
Sep 16 18:50:39.484070 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.483880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-fce992-predictor-serving-cert\""
Sep 16 18:50:39.484070 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.483933    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-fce992-dockercfg-2ssgw\""
Sep 16 18:50:39.490421 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.490400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:39.569036 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.569186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqnc\" (UniqueName: \"kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.569186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.569524 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvps\" (UniqueName: \"kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569524 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.569524 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569524 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.569695 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569760 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.569870 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.569854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.571640 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.571622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.578016 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.577976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvps\" (UniqueName: \"kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps\") pod \"isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.670003 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.669926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670003 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.669999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670191 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqnc\" (UniqueName: \"kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670191 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670191 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:50:39.670084    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-serving-cert: secret "isvc-init-fail-fce992-predictor-serving-cert" not found
Sep 16 18:50:39.670191 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:50:39.670176    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls podName:e11d2eba-f38f-47fd-bad1-9cc3cd930530 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:40.170144219 +0000 UTC m=+623.889241290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls") pod "isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530") : secret "isvc-init-fail-fce992-predictor-serving-cert" not found
Sep 16 18:50:39.670389 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670509 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670713 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.670813 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.670795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.680771 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.680748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqnc\" (UniqueName: \"kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:39.687605 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.687586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:50:39.965496 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:39.965469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:50:39.967524 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:50:39.967499    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589752fa_5569_47d2_9245_2e9325003826.slice/crio-fd8aa43e23792de965a2684b55d2c71da8959a250cc541dfdd48428be97cc92f WatchSource:0}: Error finding container fd8aa43e23792de965a2684b55d2c71da8959a250cc541dfdd48428be97cc92f: Status 404 returned error can't find the container with id fd8aa43e23792de965a2684b55d2c71da8959a250cc541dfdd48428be97cc92f
Sep 16 18:50:40.174708 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.174673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:40.177123 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.177099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") pod \"isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") " pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:40.342009 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.341978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerStarted","Data":"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"}
Sep 16 18:50:40.342180 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.342039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerStarted","Data":"fd8aa43e23792de965a2684b55d2c71da8959a250cc541dfdd48428be97cc92f"}
Sep 16 18:50:40.391959 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.391927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:40.667994 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:40.667970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:40.670131 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:50:40.670105    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11d2eba_f38f_47fd_bad1_9cc3cd930530.slice/crio-25be931ae476a597669aef87aa87325b9f68c8c13eb04b430a7c6c60e4256a19 WatchSource:0}: Error finding container 25be931ae476a597669aef87aa87325b9f68c8c13eb04b430a7c6c60e4256a19: Status 404 returned error can't find the container with id 25be931ae476a597669aef87aa87325b9f68c8c13eb04b430a7c6c60e4256a19
Sep 16 18:50:41.348112 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:41.348068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerStarted","Data":"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"}
Sep 16 18:50:41.348310 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:41.348161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerStarted","Data":"25be931ae476a597669aef87aa87325b9f68c8c13eb04b430a7c6c60e4256a19"}
Sep 16 18:50:44.363887 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.363853    2579 generic.go:356] "Generic (PLEG): container finished" podID="589752fa-5569-47d2-9245-2e9325003826" containerID="cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9" exitCode=0
Sep 16 18:50:44.364209 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.363923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerDied","Data":"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"}
Sep 16 18:50:44.365149 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.365130    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:44.365774 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.365680    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/0.log"
Sep 16 18:50:44.365774 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.365708    2579 generic.go:356] "Generic (PLEG): container finished" podID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerID="55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc" exitCode=1
Sep 16 18:50:44.365902 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:44.365791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerDied","Data":"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"}
Sep 16 18:50:45.378516 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:45.377802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/0.log"
Sep 16 18:50:45.378516 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:45.377903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerStarted","Data":"0ba508727890430c048aed0f2b5e175d0163478e5e6afd2a803da47803eba32a"}
Sep 16 18:50:49.177364 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:49.177327    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:49.177965 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:49.177601    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer" containerID="cri-o://0ba508727890430c048aed0f2b5e175d0163478e5e6afd2a803da47803eba32a" gracePeriod=30
Sep 16 18:50:53.430831 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:53.430805    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/1.log"
Sep 16 18:50:53.431260 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:53.431236    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/0.log"
Sep 16 18:50:53.431311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:53.431264    2579 generic.go:356] "Generic (PLEG): container finished" podID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerID="0ba508727890430c048aed0f2b5e175d0163478e5e6afd2a803da47803eba32a" exitCode=1
Sep 16 18:50:53.431311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:53.431290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerDied","Data":"0ba508727890430c048aed0f2b5e175d0163478e5e6afd2a803da47803eba32a"}
Sep 16 18:50:53.431405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:53.431355    2579 scope.go:117] "RemoveContainer" containerID="55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"
Sep 16 18:50:54.286013 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.285993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/1.log"
Sep 16 18:50:54.288710 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.288676    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:54.414135 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvqnc\" (UniqueName: \"kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc\") pod \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") "
Sep 16 18:50:54.414311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414177    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert\") pod \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") "
Sep 16 18:50:54.414383 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config\") pod \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") "
Sep 16 18:50:54.414383 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414355    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location\") pod \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") "
Sep 16 18:50:54.414521 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") pod \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\" (UID: \"e11d2eba-f38f-47fd-bad1-9cc3cd930530\") "
Sep 16 18:50:54.414917 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e11d2eba-f38f-47fd-bad1-9cc3cd930530" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:54.414917 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414873    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e11d2eba-f38f-47fd-bad1-9cc3cd930530" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:54.415182 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.414969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e11d2eba-f38f-47fd-bad1-9cc3cd930530" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:54.417540 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.417510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e11d2eba-f38f-47fd-bad1-9cc3cd930530" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:54.417763 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.417663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc" (OuterVolumeSpecName: "kube-api-access-tvqnc") pod "e11d2eba-f38f-47fd-bad1-9cc3cd930530" (UID: "e11d2eba-f38f-47fd-bad1-9cc3cd930530"). InnerVolumeSpecName "kube-api-access-tvqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:54.438040 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.438020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc_e11d2eba-f38f-47fd-bad1-9cc3cd930530/storage-initializer/1.log"
Sep 16 18:50:54.438676 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.438647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"
Sep 16 18:50:54.438796 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.438669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc" event={"ID":"e11d2eba-f38f-47fd-bad1-9cc3cd930530","Type":"ContainerDied","Data":"25be931ae476a597669aef87aa87325b9f68c8c13eb04b430a7c6c60e4256a19"}
Sep 16 18:50:54.482392 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.482365    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:54.492267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.492232    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-fce992-predictor-5f5cbffc4f-x2vbc"]
Sep 16 18:50:54.516517 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.516490    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.516670 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.516527    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.516670 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.516550    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.516670 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.516565    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e11d2eba-f38f-47fd-bad1-9cc3cd930530-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.516670 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.516581    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqnc\" (UniqueName: \"kubernetes.io/projected/e11d2eba-f38f-47fd-bad1-9cc3cd930530-kube-api-access-tvqnc\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:50:54.750781 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:54.750751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" path="/var/lib/kubelet/pods/e11d2eba-f38f-47fd-bad1-9cc3cd930530/volumes"
Sep 16 18:50:57.327833 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:57.327667    2579 scope.go:117] "RemoveContainer" containerID="0ba508727890430c048aed0f2b5e175d0163478e5e6afd2a803da47803eba32a"
Sep 16 18:50:57.335679 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:57.335656    2579 scope.go:117] "RemoveContainer" containerID="55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"
Sep 16 18:50:57.335945 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:50:57.335923    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc\": container with ID starting with 55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc not found: ID does not exist" containerID="55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"
Sep 16 18:50:57.336010 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:57.335958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc"} err="failed to get container status \"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc\": rpc error: code = NotFound desc = could not find container \"55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc\": container with ID starting with 55feea13e7cbfb3172c8a25d4bf29bf2ee4c7af7d8c307e8ef15432a0e24f5dc not found: ID does not exist"
Sep 16 18:50:58.467904 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:50:58.467868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerStarted","Data":"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"}
Sep 16 18:51:00.481728 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:00.481690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerStarted","Data":"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"}
Sep 16 18:51:02.495961 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:02.495929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerStarted","Data":"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"}
Sep 16 18:51:02.496351 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:02.496098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:51:02.518534 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:02.518489    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podStartSLOduration=2.65480875 podStartE2EDuration="23.518471978s" totalImagesPullingTime="17.069004217s" totalInitContainerRuntime="3.794659011s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:02.517572166 +0000 UTC m=+646.236669242" watchObservedRunningTime="2026-09-16 18:51:02.518471978 +0000 UTC m=+646.237569055"
Sep 16 18:51:03.500862 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:03.500821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:51:03.501235 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:03.500907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:51:03.502554 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:03.502522    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:03.503241 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:03.503221    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:04.505017 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:04.504975    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:04.505499 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:04.505449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:04.508720 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:04.508700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:51:05.509841 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:05.509795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:05.510267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:05.510095    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:15.510048 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:15.510000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:15.510600 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:15.510492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:25.510304 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:25.510249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:25.510758 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:25.510737    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:35.509940 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:35.509895    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:35.510405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:35.510307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:45.510649 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:45.510600    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:45.511041 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:45.510996    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:55.510441 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:55.510392    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:51:55.510932 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:51:55.510874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:05.510478 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:05.510414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:52:05.510986 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:05.510860    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:15.511168 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:15.511083    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:15.511629 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:15.511187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:24.234942 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.234910    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:52:24.235727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.235172    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" containerID="cri-o://be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932" gracePeriod=30
Sep 16 18:52:24.235727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.235172    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" containerID="cri-o://a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54" gracePeriod=30
Sep 16 18:52:24.235727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.235219    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" containerID="cri-o://5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2" gracePeriod=30
Sep 16 18:52:24.344786 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.344745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:52:24.345682 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.345657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.345811 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.345689    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.345811 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.345715    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.345811 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.345725    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.345933 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.345828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.346018 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.346004    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11d2eba-f38f-47fd-bad1-9cc3cd930530" containerName="storage-initializer"
Sep 16 18:52:24.348839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.348780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.351101 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.351075    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 18:52:24.351198 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.351122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 18:52:24.360618 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.360584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:52:24.452210 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.452179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.452373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.452222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxzh\" (UniqueName: \"kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.452373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.452269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.452373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.452311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.452373 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.452330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.505411 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.505339    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:24.553270 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.553380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.553380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxzh\" (UniqueName: \"kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.553380 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.553567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.553821 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.553788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.554111 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.554091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.554166 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.554091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.556084 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.556061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.566579 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.566557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxzh\" (UniqueName: \"kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh\") pod \"isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.659425 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.659390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:24.851374 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.851346    2579 generic.go:356] "Generic (PLEG): container finished" podID="589752fa-5569-47d2-9245-2e9325003826" containerID="5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2" exitCode=2
Sep 16 18:52:24.851533 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.851400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerDied","Data":"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"}
Sep 16 18:52:24.933807 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:24.933785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:52:24.935812 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:52:24.935791    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbcfa95_3d2f_4ddc_a0b3_73191104f93f.slice/crio-30544378ee48646d214d0cae6e759611f98b83369360faedb70f248d7e8d6ba3 WatchSource:0}: Error finding container 30544378ee48646d214d0cae6e759611f98b83369360faedb70f248d7e8d6ba3: Status 404 returned error can't find the container with id 30544378ee48646d214d0cae6e759611f98b83369360faedb70f248d7e8d6ba3
Sep 16 18:52:25.510088 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:25.510040    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:52:25.510508 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:25.510408    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:25.858377 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:25.858341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerStarted","Data":"cc7e34553d4266473d7f33272271bf5b7741afb8eeb4c178cf90538daa83dfe1"}
Sep 16 18:52:25.858821 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:25.858402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerStarted","Data":"30544378ee48646d214d0cae6e759611f98b83369360faedb70f248d7e8d6ba3"}
Sep 16 18:52:28.469444 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:52:28.469414    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589752fa_5569_47d2_9245_2e9325003826.slice/crio-be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 18:52:28.874189 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:28.874153    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerID="cc7e34553d4266473d7f33272271bf5b7741afb8eeb4c178cf90538daa83dfe1" exitCode=0
Sep 16 18:52:28.874340 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:28.874231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerDied","Data":"cc7e34553d4266473d7f33272271bf5b7741afb8eeb4c178cf90538daa83dfe1"}
Sep 16 18:52:28.877018 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:28.876997    2579 generic.go:356] "Generic (PLEG): container finished" podID="589752fa-5569-47d2-9245-2e9325003826" containerID="be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932" exitCode=0
Sep 16 18:52:28.877101 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:28.877029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerDied","Data":"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"}
Sep 16 18:52:29.505191 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.505149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:29.884405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.884375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerStarted","Data":"c078fdf2a5882a956f81e36312d2ee119819dd05a1d1927d23049891e9ae4c3e"}
Sep 16 18:52:29.884593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.884446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerStarted","Data":"bc6a44e9ed66ae7bf4a4344eb6c15e268f9058afc0277a676eb2fe4047cc283a"}
Sep 16 18:52:29.884593 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.884499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerStarted","Data":"7e5b793e4648d2698d230744c4accbae84ca9c7de4212155f5c9849c040955c6"}
Sep 16 18:52:29.884829 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.884809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:29.884978 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.884961    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:29.886221 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.886198    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:29.906777 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:29.906739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podStartSLOduration=2.086087334 podStartE2EDuration="5.906726236s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820638902s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:29.904502241 +0000 UTC m=+733.623599315" watchObservedRunningTime="2026-09-16 18:52:29.906726236 +0000 UTC m=+733.625823373"
Sep 16 18:52:30.889260 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:30.889220    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:30.889754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:30.889268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:30.890433 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:30.890403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:31.894153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:31.894106    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:31.894665 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:31.894643    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:34.505973 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:34.505932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:34.506417 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:34.506053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:35.510326 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:35.510279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:52:35.510690 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:35.510613    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:36.899770 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:36.899742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:52:36.900443 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:36.900398    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:36.900646 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:36.900616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:39.505623 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:39.505583    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:44.506049 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:44.506001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:45.509868 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:45.509811    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:52:45.510289 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:45.509956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:45.510289 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:45.510047    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:45.510289 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:45.510150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:46.900231 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:46.900190    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:46.900779 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:46.900748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:49.505716 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:49.505676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:52:54.398399 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.398368    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:54.505477 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls\") pod \"589752fa-5569-47d2-9245-2e9325003826\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") "
Sep 16 18:52:54.505477 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location\") pod \"589752fa-5569-47d2-9245-2e9325003826\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") "
Sep 16 18:52:54.505708 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fvps\" (UniqueName: \"kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps\") pod \"589752fa-5569-47d2-9245-2e9325003826\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") "
Sep 16 18:52:54.505708 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config\") pod \"589752fa-5569-47d2-9245-2e9325003826\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") "
Sep 16 18:52:54.505708 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert\") pod \"589752fa-5569-47d2-9245-2e9325003826\" (UID: \"589752fa-5569-47d2-9245-2e9325003826\") "
Sep 16 18:52:54.505851 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505754    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "589752fa-5569-47d2-9245-2e9325003826" (UID: "589752fa-5569-47d2-9245-2e9325003826"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:54.505948 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.505932    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/589752fa-5569-47d2-9245-2e9325003826-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.506023 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.506000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "589752fa-5569-47d2-9245-2e9325003826" (UID: "589752fa-5569-47d2-9245-2e9325003826"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:54.506088 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.506061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "589752fa-5569-47d2-9245-2e9325003826" (UID: "589752fa-5569-47d2-9245-2e9325003826"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:54.507800 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.507770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "589752fa-5569-47d2-9245-2e9325003826" (UID: "589752fa-5569-47d2-9245-2e9325003826"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:54.507906 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.507793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps" (OuterVolumeSpecName: "kube-api-access-5fvps") pod "589752fa-5569-47d2-9245-2e9325003826" (UID: "589752fa-5569-47d2-9245-2e9325003826"). InnerVolumeSpecName "kube-api-access-5fvps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:54.606520 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.606483    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/589752fa-5569-47d2-9245-2e9325003826-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.606520 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.606522    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvps\" (UniqueName: \"kubernetes.io/projected/589752fa-5569-47d2-9245-2e9325003826-kube-api-access-5fvps\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.606642 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.606533    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:52:54.606642 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:54.606541    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/589752fa-5569-47d2-9245-2e9325003826-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:52:55.001496 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.001441    2579 generic.go:356] "Generic (PLEG): container finished" podID="589752fa-5569-47d2-9245-2e9325003826" containerID="a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54" exitCode=0
Sep 16 18:52:55.001661 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.001499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerDied","Data":"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"}
Sep 16 18:52:55.001661 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.001539    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"
Sep 16 18:52:55.001661 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.001582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v" event={"ID":"589752fa-5569-47d2-9245-2e9325003826","Type":"ContainerDied","Data":"fd8aa43e23792de965a2684b55d2c71da8959a250cc541dfdd48428be97cc92f"}
Sep 16 18:52:55.001661 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.001628    2579 scope.go:117] "RemoveContainer" containerID="a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"
Sep 16 18:52:55.009900 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.009881    2579 scope.go:117] "RemoveContainer" containerID="5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"
Sep 16 18:52:55.017311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.017295    2579 scope.go:117] "RemoveContainer" containerID="be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"
Sep 16 18:52:55.023111 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.023091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:52:55.024833 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.024822    2579 scope.go:117] "RemoveContainer" containerID="cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"
Sep 16 18:52:55.028217 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.028196    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ccc9784d9-vxf7v"]
Sep 16 18:52:55.032144 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032130    2579 scope.go:117] "RemoveContainer" containerID="a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"
Sep 16 18:52:55.032385 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:52:55.032367    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54\": container with ID starting with a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54 not found: ID does not exist" containerID="a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"
Sep 16 18:52:55.032446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032393    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54"} err="failed to get container status \"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54\": rpc error: code = NotFound desc = could not find container \"a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54\": container with ID starting with a7a66e4ec22694b310b64ab50095415c58eb09d473d51d14fa9d0f60e64e9c54 not found: ID does not exist"
Sep 16 18:52:55.032446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032409    2579 scope.go:117] "RemoveContainer" containerID="5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"
Sep 16 18:52:55.032694 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:52:55.032678    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2\": container with ID starting with 5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2 not found: ID does not exist" containerID="5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"
Sep 16 18:52:55.032732 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032703    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2"} err="failed to get container status \"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2\": rpc error: code = NotFound desc = could not find container \"5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2\": container with ID starting with 5208feff2d580775e069b6dc25aee2ea5a27e97911f6d6d1561c03b1213c7bd2 not found: ID does not exist"
Sep 16 18:52:55.032732 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032718    2579 scope.go:117] "RemoveContainer" containerID="be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"
Sep 16 18:52:55.032953 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:52:55.032934    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932\": container with ID starting with be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932 not found: ID does not exist" containerID="be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"
Sep 16 18:52:55.033010 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932"} err="failed to get container status \"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932\": rpc error: code = NotFound desc = could not find container \"be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932\": container with ID starting with be23937d478f77d05ad22031ee7b61772bda295388b60881583d61dbd9aa4932 not found: ID does not exist"
Sep 16 18:52:55.033010 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.032975    2579 scope.go:117] "RemoveContainer" containerID="cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"
Sep 16 18:52:55.033221 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:52:55.033200    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9\": container with ID starting with cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9 not found: ID does not exist" containerID="cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"
Sep 16 18:52:55.033267 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:55.033226    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9"} err="failed to get container status \"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9\": rpc error: code = NotFound desc = could not find container \"cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9\": container with ID starting with cfaa6ef0779e7bf4f88fac587147c5e2905ae226e231740a88c02145ad89eaa9 not found: ID does not exist"
Sep 16 18:52:56.749964 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:56.749927    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="589752fa-5569-47d2-9245-2e9325003826" path="/var/lib/kubelet/pods/589752fa-5569-47d2-9245-2e9325003826/volumes"
Sep 16 18:52:56.900226 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:56.900187    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:52:56.900665 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:56.900636    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:59.629983 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.629943    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:52:59.630482 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630446    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630485    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630545    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="storage-initializer"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630554    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="storage-initializer"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy"
Sep 16 18:52:59.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy"
Sep 16 18:52:59.630913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kserve-container"
Sep 16 18:52:59.630913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630675    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="kube-rbac-proxy"
Sep 16 18:52:59.630913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.630689    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="589752fa-5569-47d2-9245-2e9325003826" containerName="agent"
Sep 16 18:52:59.633892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.633869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.636425 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.636402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 18:52:59.636562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.636489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 18:52:59.641608 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.641583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.641926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.641892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.642039 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.641978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.642039 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.642013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.642199 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.642050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlp8\" (UniqueName: \"kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.643839 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.643814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:52:59.743212 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.743356 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.743356 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlp8\" (UniqueName: \"kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.743356 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.743356 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.743813 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.743790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.744064 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.744044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.744144 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.744094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.745948 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.745925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.752188 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.752169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlp8\" (UniqueName: \"kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8\") pod \"isvc-predictive-xgboost-predictor-695f978978-cp8h8\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:52:59.945308 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:52:59.945241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:53:00.219198 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:00.219171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:53:00.221203 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:53:00.221175    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae29b658_5f9d_41fa_af7a_3e5dbaa73f89.slice/crio-00e6996a880e9a2914211e889e4ab2c3b5a804d09857cadd3e6fd14876532e70 WatchSource:0}: Error finding container 00e6996a880e9a2914211e889e4ab2c3b5a804d09857cadd3e6fd14876532e70: Status 404 returned error can't find the container with id 00e6996a880e9a2914211e889e4ab2c3b5a804d09857cadd3e6fd14876532e70
Sep 16 18:53:01.032195 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:01.032162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerStarted","Data":"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"}
Sep 16 18:53:01.032566 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:01.032219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerStarted","Data":"00e6996a880e9a2914211e889e4ab2c3b5a804d09857cadd3e6fd14876532e70"}
Sep 16 18:53:05.053590 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:05.053555    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerID="15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1" exitCode=0
Sep 16 18:53:05.053954 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:05.053630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerDied","Data":"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"}
Sep 16 18:53:06.900833 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:06.900719    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:53:06.901724 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:06.900875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:16.900242 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:16.900198    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:53:16.900742 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:16.900708    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:26.900502 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:26.900444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:53:26.900996 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:26.900974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:28.192358 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:28.192320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerStarted","Data":"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"}
Sep 16 18:53:28.192760 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:28.192409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerStarted","Data":"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"}
Sep 16 18:53:28.192760 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:28.192683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:53:28.211764 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:28.211690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podStartSLOduration=2.942857951 podStartE2EDuration="29.211673345s" totalImagesPullingTime="22.242669637s" totalInitContainerRuntime="4.026145757s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:28.210132485 +0000 UTC m=+791.929229561" watchObservedRunningTime="2026-09-16 18:53:28.211673345 +0000 UTC m=+791.930770624"
Sep 16 18:53:29.196829 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:29.196792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:53:29.197711 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:29.197688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:53:30.202298 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:30.202256    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:53:35.207572 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:35.207541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:53:35.208082 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:35.208042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:53:36.900261 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:36.900222    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:53:36.900662 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:36.900627    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:45.208182 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:45.208145    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:53:45.744159 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:45.744110    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:53:45.744586 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:45.744555    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:53:55.208314 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:55.208273    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:53:55.745292 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:55.745265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:53:55.745520 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:53:55.745502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:05.209007 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:05.208966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:54:09.445031 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:09.445002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:54:09.445407 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:09.445267    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" containerID="cri-o://7e5b793e4648d2698d230744c4accbae84ca9c7de4212155f5c9849c040955c6" gracePeriod=30
Sep 16 18:54:09.445407 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:09.445277    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" containerID="cri-o://c078fdf2a5882a956f81e36312d2ee119819dd05a1d1927d23049891e9ae4c3e" gracePeriod=30
Sep 16 18:54:09.445537 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:09.445426    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" containerID="cri-o://bc6a44e9ed66ae7bf4a4344eb6c15e268f9058afc0277a676eb2fe4047cc283a" gracePeriod=30
Sep 16 18:54:10.392269 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:10.392238    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerID="bc6a44e9ed66ae7bf4a4344eb6c15e268f9058afc0277a676eb2fe4047cc283a" exitCode=2
Sep 16 18:54:10.392442 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:10.392313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerDied","Data":"bc6a44e9ed66ae7bf4a4344eb6c15e268f9058afc0277a676eb2fe4047cc283a"}
Sep 16 18:54:11.895177 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:11.895142    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:14.413810 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:14.413771    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerID="7e5b793e4648d2698d230744c4accbae84ca9c7de4212155f5c9849c040955c6" exitCode=0
Sep 16 18:54:14.414169 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:14.413821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerDied","Data":"7e5b793e4648d2698d230744c4accbae84ca9c7de4212155f5c9849c040955c6"}
Sep 16 18:54:15.208532 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:15.208496    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:54:15.744530 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:15.744485    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:54:15.744941 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:15.744812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:16.894608 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:16.894573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:21.894754 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:21.894707    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:21.895121 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:21.894838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:25.208012 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:25.207977    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:54:25.744507 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:25.744432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:54:25.744756 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:25.744737    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:26.894926 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:26.894884    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:29.599371 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.599333    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:54:29.602345 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.602324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.604928 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.604904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 18:54:29.605031 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.604910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 18:54:29.615334 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.615312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:54:29.724198 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.724165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.724368 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.724217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.724368 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.724233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.724368 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.724266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.724368 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.724319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5tk\" (UniqueName: \"kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825079 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825079 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5tk\" (UniqueName: \"kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825259 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825259 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825259 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825259 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:29.825193    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:54:29.825479 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:29.825267    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls podName:b83ab7cd-6ec0-4edc-8fc6-6acee374307e nodeName:}" failed. No retries permitted until 2026-09-16 18:54:30.325251119 +0000 UTC m=+854.044348173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls") pod "isvc-logger-predictor-7656d4d74-8s8b4" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e") : secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:54:29.825613 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825873 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.825913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.825859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:29.834353 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:29.834332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5tk\" (UniqueName: \"kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:30.328567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:30.328526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:30.331084 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:30.331058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") pod \"isvc-logger-predictor-7656d4d74-8s8b4\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:30.512527 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:30.512497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:30.785447 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:30.785425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:54:30.787675 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:54:30.787645    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ab7cd_6ec0_4edc_8fc6_6acee374307e.slice/crio-d229983adce8e4d71836b9eb237082047f3c9960739a607c870a32a3b330a870 WatchSource:0}: Error finding container d229983adce8e4d71836b9eb237082047f3c9960739a607c870a32a3b330a870: Status 404 returned error can't find the container with id d229983adce8e4d71836b9eb237082047f3c9960739a607c870a32a3b330a870
Sep 16 18:54:31.494167 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:31.494131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerStarted","Data":"0a5ea6939dd4d349bc41143567c73fbd321b5bdf4208cce69fa7b3e436c22664"}
Sep 16 18:54:31.494310 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:31.494217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerStarted","Data":"d229983adce8e4d71836b9eb237082047f3c9960739a607c870a32a3b330a870"}
Sep 16 18:54:31.894762 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:31.894720    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:35.208794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.208713    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 16 18:54:35.515650 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.515571    2579 generic.go:356] "Generic (PLEG): container finished" podID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerID="0a5ea6939dd4d349bc41143567c73fbd321b5bdf4208cce69fa7b3e436c22664" exitCode=0
Sep 16 18:54:35.515805 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.515645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerDied","Data":"0a5ea6939dd4d349bc41143567c73fbd321b5bdf4208cce69fa7b3e436c22664"}
Sep 16 18:54:35.745052 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.745013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:5000: connect: connection refused"
Sep 16 18:54:35.745354 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.745334    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:35.745545 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.745373    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:35.745665 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:35.745609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:36.522777 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.522745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerStarted","Data":"2b6ef3bce32ffc0a719a704e7ce5d793d4705a0d65d13a8e45c34cc95c7a44b9"}
Sep 16 18:54:36.523274 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.522808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerStarted","Data":"98b5615f6fa99f1fa9f688c91a57f2802a01a7411a9198eca4a5ec1a4a1cae63"}
Sep 16 18:54:36.523274 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.522833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerStarted","Data":"457d95df643385366e252865f5060509f067fa8d641b66e97da5009eeca2493f"}
Sep 16 18:54:36.523274 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.523148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:36.523274 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.523266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:36.524804 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.524776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:54:36.544784 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.544741    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podStartSLOduration=3.585179911 podStartE2EDuration="7.544727105s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959547194s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:36.54334308 +0000 UTC m=+860.262440170" watchObservedRunningTime="2026-09-16 18:54:36.544727105 +0000 UTC m=+860.263824183"
Sep 16 18:54:36.894485 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:36.894427    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:54:37.528264 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:37.528232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:37.528645 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:37.528374    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:54:37.529511 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:37.529482    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:38.532998 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:38.532963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:54:38.533519 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:38.533497    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:39.540873 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.540823    2579 generic.go:356] "Generic (PLEG): container finished" podID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerID="c078fdf2a5882a956f81e36312d2ee119819dd05a1d1927d23049891e9ae4c3e" exitCode=0
Sep 16 18:54:39.541206 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.540871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerDied","Data":"c078fdf2a5882a956f81e36312d2ee119819dd05a1d1927d23049891e9ae4c3e"}
Sep 16 18:54:39.593993 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.593971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:39.709449 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls\") pod \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") "
Sep 16 18:54:39.709449 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location\") pod \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") "
Sep 16 18:54:39.709691 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config\") pod \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") "
Sep 16 18:54:39.709691 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert\") pod \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") "
Sep 16 18:54:39.709691 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsxzh\" (UniqueName: \"kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh\") pod \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\" (UID: \"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f\") "
Sep 16 18:54:39.709868 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" (UID: "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:39.709918 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" (UID: "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:39.709918 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.709878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" (UID: "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:39.711663 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.711642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" (UID: "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:39.711749 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.711709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh" (OuterVolumeSpecName: "kube-api-access-bsxzh") pod "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" (UID: "4cbcfa95-3d2f-4ddc-a0b3-73191104f93f"). InnerVolumeSpecName "kube-api-access-bsxzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:39.810892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.810865    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:39.810892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.810887    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:39.810892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.810897    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:39.811085 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.810906    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxzh\" (UniqueName: \"kubernetes.io/projected/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-kube-api-access-bsxzh\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:39.811085 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:39.810916    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:40.548766 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.548724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8" event={"ID":"4cbcfa95-3d2f-4ddc-a0b3-73191104f93f","Type":"ContainerDied","Data":"30544378ee48646d214d0cae6e759611f98b83369360faedb70f248d7e8d6ba3"}
Sep 16 18:54:40.549223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.548821    2579 scope.go:117] "RemoveContainer" containerID="c078fdf2a5882a956f81e36312d2ee119819dd05a1d1927d23049891e9ae4c3e"
Sep 16 18:54:40.549223 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.548843    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"
Sep 16 18:54:40.558092 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.558077    2579 scope.go:117] "RemoveContainer" containerID="bc6a44e9ed66ae7bf4a4344eb6c15e268f9058afc0277a676eb2fe4047cc283a"
Sep 16 18:54:40.565956 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.565937    2579 scope.go:117] "RemoveContainer" containerID="7e5b793e4648d2698d230744c4accbae84ca9c7de4212155f5c9849c040955c6"
Sep 16 18:54:40.572439 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.572417    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:54:40.573879 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.573857    2579 scope.go:117] "RemoveContainer" containerID="cc7e34553d4266473d7f33272271bf5b7741afb8eeb4c178cf90538daa83dfe1"
Sep 16 18:54:40.576417 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.576395    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-8485cc5cfb-wg2w8"]
Sep 16 18:54:40.750104 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:40.750065    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" path="/var/lib/kubelet/pods/4cbcfa95-3d2f-4ddc-a0b3-73191104f93f/volumes"
Sep 16 18:54:43.537735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:43.537701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:54:43.538347 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:43.538313    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:54:43.538781 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:43.538754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:45.209157 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:45.209129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:54:49.717562 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:49.717532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:54:49.718000 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:49.717755    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container" containerID="cri-o://5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd" gracePeriod=30
Sep 16 18:54:49.718000 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:49.717820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kube-rbac-proxy" containerID="cri-o://ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4" gracePeriod=30
Sep 16 18:54:50.203205 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:50.203167    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 18:54:50.601674 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:50.601644    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerID="ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4" exitCode=2
Sep 16 18:54:50.601834 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:50.601700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerDied","Data":"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"}
Sep 16 18:54:53.538247 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:53.538202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:54:53.538786 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:53.538765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:54:54.170515 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.170488    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:54:54.220596 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls\") pod \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") "
Sep 16 18:54:54.220596 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location\") pod \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") "
Sep 16 18:54:54.220783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnlp8\" (UniqueName: \"kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8\") pod \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") "
Sep 16 18:54:54.220783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config\") pod \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") "
Sep 16 18:54:54.220783 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert\") pod \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\" (UID: \"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89\") "
Sep 16 18:54:54.221004 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.220950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" (UID: "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:54.221116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.221009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" (UID: "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:54.221116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.221072    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" (UID: "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:54.222777 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.222753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" (UID: "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:54.222915 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.222895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8" (OuterVolumeSpecName: "kube-api-access-lnlp8") pod "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" (UID: "ae29b658-5f9d-41fa-af7a-3e5dbaa73f89"). InnerVolumeSpecName "kube-api-access-lnlp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:54.322099 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.322062    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:54.322099 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.322089    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:54.322099 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.322098    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:54.322311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.322106    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:54.322311 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.322119    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlp8\" (UniqueName: \"kubernetes.io/projected/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89-kube-api-access-lnlp8\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:54:54.622002 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.621967    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerID="5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd" exitCode=0
Sep 16 18:54:54.622366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.622026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"
Sep 16 18:54:54.622366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.622048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerDied","Data":"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"}
Sep 16 18:54:54.622366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.622109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8" event={"ID":"ae29b658-5f9d-41fa-af7a-3e5dbaa73f89","Type":"ContainerDied","Data":"00e6996a880e9a2914211e889e4ab2c3b5a804d09857cadd3e6fd14876532e70"}
Sep 16 18:54:54.622366 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.622137    2579 scope.go:117] "RemoveContainer" containerID="ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"
Sep 16 18:54:54.632272 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.632194    2579 scope.go:117] "RemoveContainer" containerID="5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"
Sep 16 18:54:54.641496 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.641473    2579 scope.go:117] "RemoveContainer" containerID="15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"
Sep 16 18:54:54.649881 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.649859    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:54:54.650281 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.650266    2579 scope.go:117] "RemoveContainer" containerID="ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"
Sep 16 18:54:54.650580 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:54.650564    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4\": container with ID starting with ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4 not found: ID does not exist" containerID="ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"
Sep 16 18:54:54.650640 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.650589    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4"} err="failed to get container status \"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4\": rpc error: code = NotFound desc = could not find container \"ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4\": container with ID starting with ed54264471caba001663c5fea7286020f0e36d8782f86950989e59fa02a02ea4 not found: ID does not exist"
Sep 16 18:54:54.650640 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.650603    2579 scope.go:117] "RemoveContainer" containerID="5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"
Sep 16 18:54:54.650878 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:54.650861    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd\": container with ID starting with 5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd not found: ID does not exist" containerID="5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"
Sep 16 18:54:54.650913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.650884    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd"} err="failed to get container status \"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd\": rpc error: code = NotFound desc = could not find container \"5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd\": container with ID starting with 5fe23bcd30fdfce929ae7eba1de7ca0eb27354065a290e3be54d6e1ccbc877fd not found: ID does not exist"
Sep 16 18:54:54.650913 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.650898    2579 scope.go:117] "RemoveContainer" containerID="15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"
Sep 16 18:54:54.651163 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:54.651139    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1\": container with ID starting with 15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1 not found: ID does not exist" containerID="15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"
Sep 16 18:54:54.651240 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.651167    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1"} err="failed to get container status \"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1\": rpc error: code = NotFound desc = could not find container \"15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1\": container with ID starting with 15e1fe3d2fd4edc1c7c734d876be34c7cc33855ce27b93b4240c22db19b3ddf1 not found: ID does not exist"
Sep 16 18:54:54.657586 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.657568    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-cp8h8"]
Sep 16 18:54:54.750720 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.750689    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" path="/var/lib/kubelet/pods/ae29b658-5f9d-41fa-af7a-3e5dbaa73f89/volumes"
Sep 16 18:54:54.908388 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908320    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:54:54.908710 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="storage-initializer"
Sep 16 18:54:54.908757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908712    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="storage-initializer"
Sep 16 18:54:54.908757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container"
Sep 16 18:54:54.908757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908735    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container"
Sep 16 18:54:54.908757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908757 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908760    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908765    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="storage-initializer"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908777    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="storage-initializer"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908783    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908793    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908798    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908865    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kserve-container"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908880    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="kube-rbac-proxy"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908884    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae29b658-5f9d-41fa-af7a-3e5dbaa73f89" containerName="kserve-container"
Sep 16 18:54:54.908890 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.908892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbcfa95-3d2f-4ddc-a0b3-73191104f93f" containerName="agent"
Sep 16 18:54:54.913820 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.913798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:54.916153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.916131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 18:54:54.916277 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.916205    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 18:54:54.925472 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.924056    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:54:54.929825 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.929692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmq7\" (UniqueName: \"kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:54.929932 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.929887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:54.929997 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.929934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:54.929997 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.929982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:54.930089 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:54.930002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030497 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030613 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030613 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030731 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:55.030681    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:54:55.030731 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030832 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:54:55.030740    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls podName:b123b0d4-9a4a-4a27-a2a4-62b44137a9c4 nodeName:}" failed. No retries permitted until 2026-09-16 18:54:55.530720292 +0000 UTC m=+879.249817361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:54:55.030832 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmq7\" (UniqueName: \"kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.030936 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.030880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.031171 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.031153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.031225 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.031187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.039321 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.039301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmq7\" (UniqueName: \"kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.534777 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.534745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.537290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.537263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:55.830556 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:55.830528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:54:56.103886 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:56.103864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:54:56.105938 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:54:56.105910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb123b0d4_9a4a_4a27_a2a4_62b44137a9c4.slice/crio-38964a870ba57ca72aa1d0084a66e6aae3fa5fd825bf85a92384f407d8b2264f WatchSource:0}: Error finding container 38964a870ba57ca72aa1d0084a66e6aae3fa5fd825bf85a92384f407d8b2264f: Status 404 returned error can't find the container with id 38964a870ba57ca72aa1d0084a66e6aae3fa5fd825bf85a92384f407d8b2264f
Sep 16 18:54:56.634059 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:56.634025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerStarted","Data":"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"}
Sep 16 18:54:56.634234 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:54:56.634080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerStarted","Data":"38964a870ba57ca72aa1d0084a66e6aae3fa5fd825bf85a92384f407d8b2264f"}
Sep 16 18:55:00.657396 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:00.657362    2579 generic.go:356] "Generic (PLEG): container finished" podID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerID="50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41" exitCode=0
Sep 16 18:55:00.657785 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:00.657437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerDied","Data":"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"}
Sep 16 18:55:01.664740 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:01.664703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerStarted","Data":"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"}
Sep 16 18:55:01.665119 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:01.664772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerStarted","Data":"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"}
Sep 16 18:55:01.665119 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:01.664976    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:55:01.686663 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:01.686617    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podStartSLOduration=3.749667382 podStartE2EDuration="7.686603949s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.936936567s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:01.685351945 +0000 UTC m=+885.404449021" watchObservedRunningTime="2026-09-16 18:55:01.686603949 +0000 UTC m=+885.405701026"
Sep 16 18:55:02.669796 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:02.669762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:55:02.671037 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:02.671007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:03.538238 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:03.538197    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:55:03.538717 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:03.538686    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:03.674393 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:03.674351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:08.678830 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:08.678763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:55:08.679326 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:08.679299    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:13.538686 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:13.538640    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:55:13.539192 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:13.539172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:16.739143 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:16.739110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:55:16.741755 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:16.741736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 18:55:18.679220 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:18.679178    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:23.538258 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:23.538217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:55:23.538761 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:23.538738    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:28.680090 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:28.680053    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:33.538787 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:33.538744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:55:33.539295 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:33.539272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:38.679975 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:38.679937    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:43.539065 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:43.539018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:55:43.541509 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:43.539488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:48.679479 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:48.679419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:55:53.539613 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:53.539579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:55:53.539980 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:53.539847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:55:58.679212 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:55:58.679175    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:56:04.753016 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.752976    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:56:04.753494 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.753335    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" containerID="cri-o://457d95df643385366e252865f5060509f067fa8d641b66e97da5009eeca2493f" gracePeriod=30
Sep 16 18:56:04.753597 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.753493    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" containerID="cri-o://2b6ef3bce32ffc0a719a704e7ce5d793d4705a0d65d13a8e45c34cc95c7a44b9" gracePeriod=30
Sep 16 18:56:04.753597 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.753547    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" containerID="cri-o://98b5615f6fa99f1fa9f688c91a57f2802a01a7411a9198eca4a5ec1a4a1cae63" gracePeriod=30
Sep 16 18:56:04.902029 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.901997    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:56:04.905818 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.905796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:04.908357 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.908325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 18:56:04.908509 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.908379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 18:56:04.916134 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.916105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:56:04.969918 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.969882    2579 generic.go:356] "Generic (PLEG): container finished" podID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerID="98b5615f6fa99f1fa9f688c91a57f2802a01a7411a9198eca4a5ec1a4a1cae63" exitCode=2
Sep 16 18:56:04.970071 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.969960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerDied","Data":"98b5615f6fa99f1fa9f688c91a57f2802a01a7411a9198eca4a5ec1a4a1cae63"}
Sep 16 18:56:04.996858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.996824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:04.996858 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.996858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:04.997063 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.996894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:04.997063 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.996961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65xt\" (UniqueName: \"kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:04.997063 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:04.996997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098370 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098550 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65xt\" (UniqueName: \"kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098550 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098550 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098741 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.098990 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.098960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.099163 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.099144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.099228 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.099171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.100916 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.100896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.107375 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.107356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65xt\" (UniqueName: \"kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt\") pod \"isvc-autogluon-v1-predictor-574b5b7495-r5d6n\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.218727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.218691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:56:05.492888 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.492862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:56:05.494387 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:56:05.494360    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e56e8f3_7da3_4478_ab20_15fb124ee83e.slice/crio-7ebb861596708fbdca16c13954e59344444950d16c1509d45eef355756aa7a1d WatchSource:0}: Error finding container 7ebb861596708fbdca16c13954e59344444950d16c1509d45eef355756aa7a1d: Status 404 returned error can't find the container with id 7ebb861596708fbdca16c13954e59344444950d16c1509d45eef355756aa7a1d
Sep 16 18:56:05.977408 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.977368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerStarted","Data":"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"}
Sep 16 18:56:05.977887 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:05.977435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerStarted","Data":"7ebb861596708fbdca16c13954e59344444950d16c1509d45eef355756aa7a1d"}
Sep 16 18:56:08.533477 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:08.533424    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:08.680266 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:08.680224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:56:09.998173 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:09.998141    2579 generic.go:356] "Generic (PLEG): container finished" podID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerID="457d95df643385366e252865f5060509f067fa8d641b66e97da5009eeca2493f" exitCode=0
Sep 16 18:56:09.998561 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:09.998189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerDied","Data":"457d95df643385366e252865f5060509f067fa8d641b66e97da5009eeca2493f"}
Sep 16 18:56:13.533306 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:13.533266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:13.538620 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:13.538592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:56:13.538986 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:13.538969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:18.533722 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:18.533674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:18.534083 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:18.533810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:56:18.679609 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:18.679582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:56:23.533809 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:23.533767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:23.539162 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:23.539122    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:56:23.539495 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:23.539448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:24.981560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:24.981451    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:56:24.981919 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:24.981765    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" containerID="cri-o://bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1" gracePeriod=30
Sep 16 18:56:24.981919 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:24.981790    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kube-rbac-proxy" containerID="cri-o://8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7" gracePeriod=30
Sep 16 18:56:26.074760 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:26.074725    2579 generic.go:356] "Generic (PLEG): container finished" podID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerID="8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7" exitCode=2
Sep 16 18:56:26.075116 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:26.074798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerDied","Data":"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"}
Sep 16 18:56:28.533971 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:28.533937    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:28.674518 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:28.674470    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 18:56:28.679187 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:28.679162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 18:56:30.625636 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.625609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:56:30.723996 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.723963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kmq7\" (UniqueName: \"kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7\") pod \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") "
Sep 16 18:56:30.724157 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") pod \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") "
Sep 16 18:56:30.724157 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location\") pod \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") "
Sep 16 18:56:30.724157 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config\") pod \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") "
Sep 16 18:56:30.724321 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724171    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert\") pod \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\" (UID: \"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4\") "
Sep 16 18:56:30.724442 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:30.724579 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:30.724638 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.724569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:30.726318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.726298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:30.726318 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.726311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7" (OuterVolumeSpecName: "kube-api-access-2kmq7") pod "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" (UID: "b123b0d4-9a4a-4a27-a2a4-62b44137a9c4"). InnerVolumeSpecName "kube-api-access-2kmq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:30.825642 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.825607    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:30.825642 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.825637    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:30.825815 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.825651    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmq7\" (UniqueName: \"kubernetes.io/projected/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kube-api-access-2kmq7\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:30.825815 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.825663    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:30.825815 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:30.825677    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:31.099614 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.099581    2579 generic.go:356] "Generic (PLEG): container finished" podID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerID="bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1" exitCode=0
Sep 16 18:56:31.099794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.099649    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"
Sep 16 18:56:31.099794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.099669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerDied","Data":"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"}
Sep 16 18:56:31.099794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.099729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx" event={"ID":"b123b0d4-9a4a-4a27-a2a4-62b44137a9c4","Type":"ContainerDied","Data":"38964a870ba57ca72aa1d0084a66e6aae3fa5fd825bf85a92384f407d8b2264f"}
Sep 16 18:56:31.099794 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.099756    2579 scope.go:117] "RemoveContainer" containerID="8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"
Sep 16 18:56:31.108252 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.108238    2579 scope.go:117] "RemoveContainer" containerID="bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"
Sep 16 18:56:31.116153 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.116136    2579 scope.go:117] "RemoveContainer" containerID="50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"
Sep 16 18:56:31.123857 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.123837    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:56:31.123908 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.123867    2579 scope.go:117] "RemoveContainer" containerID="8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"
Sep 16 18:56:31.124126 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:56:31.124109    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7\": container with ID starting with 8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7 not found: ID does not exist" containerID="8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"
Sep 16 18:56:31.124186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.124134    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7"} err="failed to get container status \"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7\": rpc error: code = NotFound desc = could not find container \"8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7\": container with ID starting with 8df66cc738e2ac67a28e6958b34335c39d8aaad250e284490b8a5542dbcbe1b7 not found: ID does not exist"
Sep 16 18:56:31.124186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.124150    2579 scope.go:117] "RemoveContainer" containerID="bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"
Sep 16 18:56:31.124389 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:56:31.124373    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1\": container with ID starting with bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1 not found: ID does not exist" containerID="bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"
Sep 16 18:56:31.124429 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.124394    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1"} err="failed to get container status \"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1\": rpc error: code = NotFound desc = could not find container \"bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1\": container with ID starting with bece92fbac4dbb9467fad7c47548508d73a9d012eb81531117a2e7b2931de2f1 not found: ID does not exist"
Sep 16 18:56:31.124429 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.124409    2579 scope.go:117] "RemoveContainer" containerID="50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"
Sep 16 18:56:31.124624 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:56:31.124610    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41\": container with ID starting with 50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41 not found: ID does not exist" containerID="50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"
Sep 16 18:56:31.124664 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.124627    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41"} err="failed to get container status \"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41\": rpc error: code = NotFound desc = could not find container \"50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41\": container with ID starting with 50ee5a2215b4d29435615975a4e422c3de0252370c016959f9c6279ed0bf9e41 not found: ID does not exist"
Sep 16 18:56:31.130915 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:31.130896    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jrgsx"]
Sep 16 18:56:32.748842 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:32.748811    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" path="/var/lib/kubelet/pods/b123b0d4-9a4a-4a27-a2a4-62b44137a9c4/volumes"
Sep 16 18:56:33.533277 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:33.533235    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:56:33.538583 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:33.538561    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:56:33.538679 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:33.538663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:56:33.538996 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:33.538974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:33.539076 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:33.539065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:56:35.122655 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.122626    2579 generic.go:356] "Generic (PLEG): container finished" podID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerID="2b6ef3bce32ffc0a719a704e7ce5d793d4705a0d65d13a8e45c34cc95c7a44b9" exitCode=0
Sep 16 18:56:35.123055 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.122684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerDied","Data":"2b6ef3bce32ffc0a719a704e7ce5d793d4705a0d65d13a8e45c34cc95c7a44b9"}
Sep 16 18:56:35.397852 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.397826    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:56:35.559789 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.559757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") pod \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") "
Sep 16 18:56:35.559939 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.559817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config\") pod \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") "
Sep 16 18:56:35.559939 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.559857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert\") pod \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") "
Sep 16 18:56:35.559939 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.559873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr5tk\" (UniqueName: \"kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk\") pod \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") "
Sep 16 18:56:35.559939 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.559895    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location\") pod \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\" (UID: \"b83ab7cd-6ec0-4edc-8fc6-6acee374307e\") "
Sep 16 18:56:35.560265 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.560243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b83ab7cd-6ec0-4edc-8fc6-6acee374307e" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:35.560265 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.560252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b83ab7cd-6ec0-4edc-8fc6-6acee374307e" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:35.560371 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.560251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b83ab7cd-6ec0-4edc-8fc6-6acee374307e" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:35.562040 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.562021    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b83ab7cd-6ec0-4edc-8fc6-6acee374307e" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:35.562097 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.562073    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk" (OuterVolumeSpecName: "kube-api-access-gr5tk") pod "b83ab7cd-6ec0-4edc-8fc6-6acee374307e" (UID: "b83ab7cd-6ec0-4edc-8fc6-6acee374307e"). InnerVolumeSpecName "kube-api-access-gr5tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:35.661503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.661403    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:35.661503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.661432    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5tk\" (UniqueName: \"kubernetes.io/projected/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-api-access-gr5tk\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:35.661503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.661443    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:35.661503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.661472    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:35.661503 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:35.661484    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b83ab7cd-6ec0-4edc-8fc6-6acee374307e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:56:36.134938 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.134893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4" event={"ID":"b83ab7cd-6ec0-4edc-8fc6-6acee374307e","Type":"ContainerDied","Data":"d229983adce8e4d71836b9eb237082047f3c9960739a607c870a32a3b330a870"}
Sep 16 18:56:36.135405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.135035    2579 scope.go:117] "RemoveContainer" containerID="2b6ef3bce32ffc0a719a704e7ce5d793d4705a0d65d13a8e45c34cc95c7a44b9"
Sep 16 18:56:36.135405 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.135228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"
Sep 16 18:56:36.137048 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.137025    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerID="92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6" exitCode=0
Sep 16 18:56:36.137151 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.137083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerDied","Data":"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"}
Sep 16 18:56:36.138431 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.138416    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:36.145838 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.145820    2579 scope.go:117] "RemoveContainer" containerID="98b5615f6fa99f1fa9f688c91a57f2802a01a7411a9198eca4a5ec1a4a1cae63"
Sep 16 18:56:36.154442 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.154427    2579 scope.go:117] "RemoveContainer" containerID="457d95df643385366e252865f5060509f067fa8d641b66e97da5009eeca2493f"
Sep 16 18:56:36.161985 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.161970    2579 scope.go:117] "RemoveContainer" containerID="0a5ea6939dd4d349bc41143567c73fbd321b5bdf4208cce69fa7b3e436c22664"
Sep 16 18:56:36.178321 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.178299    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:56:36.186595 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.186576    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7656d4d74-8s8b4"]
Sep 16 18:56:36.750825 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:56:36.750791    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" path="/var/lib/kubelet/pods/b83ab7cd-6ec0-4edc-8fc6-6acee374307e/volumes"
Sep 16 18:58:38.757727 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.757687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerStarted","Data":"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"}
Sep 16 18:58:38.758184 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.757755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerStarted","Data":"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"}
Sep 16 18:58:38.758184 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.757922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:58:38.758184 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.758057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:58:38.759472 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.759428    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:58:38.780785 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:38.780735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podStartSLOduration=3.424972198 podStartE2EDuration="2m34.780722148s" totalImagesPullingTime="2m1.767243811s" totalInitContainerRuntime="29.588506139s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:38.77878951 +0000 UTC m=+1102.497886588" watchObservedRunningTime="2026-09-16 18:58:38.780722148 +0000 UTC m=+1102.499819225"
Sep 16 18:58:39.761998 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:39.761957    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:58:44.767324 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:44.767292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:58:44.767908 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:44.767881    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:58:54.768657 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:58:54.768630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:59:11.436643 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:11.436608    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:59:11.437118 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:11.436850    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container" containerID="cri-o://d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47" gracePeriod=30
Sep 16 18:59:11.437118 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:11.436914    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kube-rbac-proxy" containerID="cri-o://4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378" gracePeriod=30
Sep 16 18:59:11.904389 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:11.904355    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerID="4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378" exitCode=2
Sep 16 18:59:11.904546 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:11.904436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerDied","Data":"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"}
Sep 16 18:59:12.816659 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.816633    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:59:12.847702 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.847646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls\") pod \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") "
Sep 16 18:59:12.847702 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.847690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert\") pod \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") "
Sep 16 18:59:12.847892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.847717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location\") pod \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") "
Sep 16 18:59:12.847892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.847748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q65xt\" (UniqueName: \"kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt\") pod \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") "
Sep 16 18:59:12.847892 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.847823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config\") pod \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\" (UID: \"0e56e8f3-7da3-4478-ab20-15fb124ee83e\") "
Sep 16 18:59:12.848041 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.848016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e56e8f3-7da3-4478-ab20-15fb124ee83e" (UID: "0e56e8f3-7da3-4478-ab20-15fb124ee83e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:12.848367 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.848299    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e56e8f3-7da3-4478-ab20-15fb124ee83e" (UID: "0e56e8f3-7da3-4478-ab20-15fb124ee83e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:12.849847 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.849821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e56e8f3-7da3-4478-ab20-15fb124ee83e" (UID: "0e56e8f3-7da3-4478-ab20-15fb124ee83e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:12.850040 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.850018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt" (OuterVolumeSpecName: "kube-api-access-q65xt") pod "0e56e8f3-7da3-4478-ab20-15fb124ee83e" (UID: "0e56e8f3-7da3-4478-ab20-15fb124ee83e"). InnerVolumeSpecName "kube-api-access-q65xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:12.852522 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.852496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e56e8f3-7da3-4478-ab20-15fb124ee83e" (UID: "0e56e8f3-7da3-4478-ab20-15fb124ee83e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:12.910416 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.910385    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerID="d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47" exitCode=0
Sep 16 18:59:12.910536 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.910426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerDied","Data":"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"}
Sep 16 18:59:12.910536 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.910436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"
Sep 16 18:59:12.910536 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.910514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n" event={"ID":"0e56e8f3-7da3-4478-ab20-15fb124ee83e","Type":"ContainerDied","Data":"7ebb861596708fbdca16c13954e59344444950d16c1509d45eef355756aa7a1d"}
Sep 16 18:59:12.910697 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.910556    2579 scope.go:117] "RemoveContainer" containerID="4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"
Sep 16 18:59:12.919305 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.919284    2579 scope.go:117] "RemoveContainer" containerID="d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"
Sep 16 18:59:12.926654 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.926635    2579 scope.go:117] "RemoveContainer" containerID="92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"
Sep 16 18:59:12.933843 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.933828    2579 scope.go:117] "RemoveContainer" containerID="4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"
Sep 16 18:59:12.934116 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:59:12.934097    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378\": container with ID starting with 4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378 not found: ID does not exist" containerID="4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"
Sep 16 18:59:12.934165 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.934123    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378"} err="failed to get container status \"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378\": rpc error: code = NotFound desc = could not find container \"4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378\": container with ID starting with 4893ad77e907cd5287749d416c7149e5efa9a24e358928caec28aab10742f378 not found: ID does not exist"
Sep 16 18:59:12.934165 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.934139    2579 scope.go:117] "RemoveContainer" containerID="d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"
Sep 16 18:59:12.934393 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:59:12.934378    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47\": container with ID starting with d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47 not found: ID does not exist" containerID="d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"
Sep 16 18:59:12.934450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.934396    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47"} err="failed to get container status \"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47\": rpc error: code = NotFound desc = could not find container \"d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47\": container with ID starting with d2289ddd34c6be53ab374d234c13f0330692a8eaea00c733b4c8f2b3d09b5e47 not found: ID does not exist"
Sep 16 18:59:12.934450 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.934407    2579 scope.go:117] "RemoveContainer" containerID="92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"
Sep 16 18:59:12.934666 ip-10-0-136-81 kubenswrapper[2579]: E0916 18:59:12.934650    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6\": container with ID starting with 92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6 not found: ID does not exist" containerID="92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"
Sep 16 18:59:12.934704 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.934669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6"} err="failed to get container status \"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6\": rpc error: code = NotFound desc = could not find container \"92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6\": container with ID starting with 92a8d27f0b27e5bf7169635f16132f2dd57677bee60f04ed9a7d503ad440c7f6 not found: ID does not exist"
Sep 16 18:59:12.944052 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.944034    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:59:12.948944 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.948919    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e56e8f3-7da3-4478-ab20-15fb124ee83e-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.948944 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.948944    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.949075 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.948958    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.949075 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.948972    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65xt\" (UniqueName: \"kubernetes.io/projected/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-api-access-q65xt\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.949075 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.948985    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e56e8f3-7da3-4478-ab20-15fb124ee83e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 18:59:12.951235 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:12.951215    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-r5d6n"]
Sep 16 18:59:14.748851 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:14.748819    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" path="/var/lib/kubelet/pods/0e56e8f3-7da3-4478-ab20-15fb124ee83e/volumes"
Sep 16 18:59:16.717186 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 18:59:16.717567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717527    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717540    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="storage-initializer"
Sep 16 18:59:16.717567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717554    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="storage-initializer"
Sep 16 18:59:16.717567 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717566    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="storage-initializer"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717572    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="storage-initializer"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717584    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717589    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717597    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717607    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="storage-initializer"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717612    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="storage-initializer"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717619    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717631    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717636    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717647    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717652    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717656    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717719    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kserve-container"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kube-rbac-proxy"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="agent"
Sep 16 18:59:16.717735 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717741    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83ab7cd-6ec0-4edc-8fc6-6acee374307e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.718379 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717746    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kserve-container"
Sep 16 18:59:16.718379 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717752    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b123b0d4-9a4a-4a27-a2a4-62b44137a9c4" containerName="kserve-container"
Sep 16 18:59:16.718379 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.717758    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e56e8f3-7da3-4478-ab20-15fb124ee83e" containerName="kube-rbac-proxy"
Sep 16 18:59:16.720057 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.720038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.724402 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.724380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 16 18:59:16.725346 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.725329    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:59:16.725572 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.725548    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:59:16.726514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.726298    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:59:16.726514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.726351    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 16 18:59:16.726514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.726387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 18:59:16.726514 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.726477    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:59:16.742952 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.742929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 18:59:16.775756 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.775722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.775959 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.775789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.775959 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.775849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.775959 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.775874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtqj\" (UniqueName: \"kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.775959 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.775927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876285 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtqj\" (UniqueName: \"kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876446 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.876846 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.876825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.877040 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.877021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.877101 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.877082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.878770 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.878749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:16.885563 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:16.885543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtqj\" (UniqueName: \"kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-wps2s\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:17.030787 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:17.030708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:17.306909 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:17.306886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 18:59:17.309005 ip-10-0-136-81 kubenswrapper[2579]: W0916 18:59:17.308979    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf7e633_76c7_41e4_80e1_41b861b7d26a.slice/crio-c46de88d2f0e0cddf7ae0b0ebce129cc07f829b6009fede5eb8e37866979e10f WatchSource:0}: Error finding container c46de88d2f0e0cddf7ae0b0ebce129cc07f829b6009fede5eb8e37866979e10f: Status 404 returned error can't find the container with id c46de88d2f0e0cddf7ae0b0ebce129cc07f829b6009fede5eb8e37866979e10f
Sep 16 18:59:17.933241 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:17.933205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerStarted","Data":"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"}
Sep 16 18:59:17.933694 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:17.933262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerStarted","Data":"c46de88d2f0e0cddf7ae0b0ebce129cc07f829b6009fede5eb8e37866979e10f"}
Sep 16 18:59:47.055290 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:47.055249    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerID="a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd" exitCode=0
Sep 16 18:59:47.055725 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:47.055322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerDied","Data":"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"}
Sep 16 18:59:48.062406 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:48.062374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerStarted","Data":"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"}
Sep 16 18:59:48.062802 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:48.062443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerStarted","Data":"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"}
Sep 16 18:59:48.062802 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:48.062670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:48.085389 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:48.085342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" podStartSLOduration=3.399110286 podStartE2EDuration="32.085326609s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.686216323s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:48.084255202 +0000 UTC m=+1171.803352280" watchObservedRunningTime="2026-09-16 18:59:48.085326609 +0000 UTC m=+1171.804423685"
Sep 16 18:59:49.066480 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:49.066437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 18:59:55.075560 ip-10-0-136-81 kubenswrapper[2579]: I0916 18:59:55.075531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 19:00:16.770395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:00:16.770366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:00:16.773118 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:00:16.773097    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:01:22.749132 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:22.749066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 19:01:32.896323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:32.896286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 19:01:32.896769 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:32.896680    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kserve-container" containerID="cri-o://9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b" gracePeriod=30
Sep 16 19:01:32.896845 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:32.896778    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kube-rbac-proxy" containerID="cri-o://5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8" gracePeriod=30
Sep 16 19:01:33.522419 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:33.522383    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerID="5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8" exitCode=2
Sep 16 19:01:33.522645 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:33.522473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerDied","Data":"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"}
Sep 16 19:01:34.231618 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.231598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 19:01:34.286688 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls\") pod \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") "
Sep 16 19:01:34.286817 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location\") pod \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") "
Sep 16 19:01:34.286817 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config\") pod \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") "
Sep 16 19:01:34.286943 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxtqj\" (UniqueName: \"kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj\") pod \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") "
Sep 16 19:01:34.287006 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286982    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcf7e633-76c7-41e4-80e1-41b861b7d26a" (UID: "fcf7e633-76c7-41e4-80e1-41b861b7d26a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:34.287006 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.286985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert\") pod \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\" (UID: \"fcf7e633-76c7-41e4-80e1-41b861b7d26a\") "
Sep 16 19:01:34.287301 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.287277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcf7e633-76c7-41e4-80e1-41b861b7d26a" (UID: "fcf7e633-76c7-41e4-80e1-41b861b7d26a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:34.287418 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.287289    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:01:34.288884 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.288862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcf7e633-76c7-41e4-80e1-41b861b7d26a" (UID: "fcf7e633-76c7-41e4-80e1-41b861b7d26a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:34.289122 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.289071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj" (OuterVolumeSpecName: "kube-api-access-qxtqj") pod "fcf7e633-76c7-41e4-80e1-41b861b7d26a" (UID: "fcf7e633-76c7-41e4-80e1-41b861b7d26a"). InnerVolumeSpecName "kube-api-access-qxtqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:34.291600 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.291577    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcf7e633-76c7-41e4-80e1-41b861b7d26a" (UID: "fcf7e633-76c7-41e4-80e1-41b861b7d26a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:34.388262 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.388230    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtqj\" (UniqueName: \"kubernetes.io/projected/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kube-api-access-qxtqj\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:01:34.388262 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.388262    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcf7e633-76c7-41e4-80e1-41b861b7d26a-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:01:34.388262 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.388272    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcf7e633-76c7-41e4-80e1-41b861b7d26a-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:01:34.388439 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.388280    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcf7e633-76c7-41e4-80e1-41b861b7d26a-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:01:34.528671 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.528641    2579 generic.go:356] "Generic (PLEG): container finished" podID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerID="9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b" exitCode=0
Sep 16 19:01:34.528822 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.528700    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"
Sep 16 19:01:34.528822 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.528722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerDied","Data":"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"}
Sep 16 19:01:34.528822 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.528795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s" event={"ID":"fcf7e633-76c7-41e4-80e1-41b861b7d26a","Type":"ContainerDied","Data":"c46de88d2f0e0cddf7ae0b0ebce129cc07f829b6009fede5eb8e37866979e10f"}
Sep 16 19:01:34.528993 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.528831    2579 scope.go:117] "RemoveContainer" containerID="5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"
Sep 16 19:01:34.537428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.537411    2579 scope.go:117] "RemoveContainer" containerID="9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"
Sep 16 19:01:34.545619 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.545567    2579 scope.go:117] "RemoveContainer" containerID="a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"
Sep 16 19:01:34.553018 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.552970    2579 scope.go:117] "RemoveContainer" containerID="5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"
Sep 16 19:01:34.553419 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:01:34.553368    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8\": container with ID starting with 5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8 not found: ID does not exist" containerID="5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"
Sep 16 19:01:34.553419 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.553407    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8"} err="failed to get container status \"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8\": rpc error: code = NotFound desc = could not find container \"5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8\": container with ID starting with 5114818f0c31894cf4a5925dd5c6ba3460833578192494eba579e51584db31e8 not found: ID does not exist"
Sep 16 19:01:34.553602 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.553431    2579 scope.go:117] "RemoveContainer" containerID="9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"
Sep 16 19:01:34.553809 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:01:34.553783    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b\": container with ID starting with 9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b not found: ID does not exist" containerID="9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"
Sep 16 19:01:34.553863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.553816    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b"} err="failed to get container status \"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b\": rpc error: code = NotFound desc = could not find container \"9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b\": container with ID starting with 9f83c103e7937a173813afc301ff48192af6c8e51888eade21d028460431ec8b not found: ID does not exist"
Sep 16 19:01:34.553863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.553834    2579 scope.go:117] "RemoveContainer" containerID="a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"
Sep 16 19:01:34.554098 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:01:34.554080    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd\": container with ID starting with a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd not found: ID does not exist" containerID="a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"
Sep 16 19:01:34.554167 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.554133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd"} err="failed to get container status \"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd\": rpc error: code = NotFound desc = could not find container \"a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd\": container with ID starting with a49cf5f5fc688e92bbc9c86fc180783e9e578ae1ac762b82c501464a0dcd96fd not found: ID does not exist"
Sep 16 19:01:34.555360 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.555342    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 19:01:34.562797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.562776    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-wps2s"]
Sep 16 19:01:34.749059 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:34.749030    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" path="/var/lib/kubelet/pods/fcf7e633-76c7-41e4-80e1-41b861b7d26a/volumes"
Sep 16 19:01:38.080657 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080622    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080940    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="storage-initializer"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080951    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="storage-initializer"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080963    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kserve-container"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080969    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kserve-container"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080988    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kube-rbac-proxy"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.080994    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kube-rbac-proxy"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.081043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kube-rbac-proxy"
Sep 16 19:01:38.081050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.081053    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf7e633-76c7-41e4-80e1-41b861b7d26a" containerName="kserve-container"
Sep 16 19:01:38.084793 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.084777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.087606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.087576    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:01:38.087739 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.087627    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:01:38.087739 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.087627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:01:38.087739 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.087628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 16 19:01:38.088747 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.088728    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:01:38.088747 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.088736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:01:38.088747 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.088735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 16 19:01:38.105437 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.105411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:01:38.219380 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.219344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.219569 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.219435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.219569 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.219532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcwt\" (UniqueName: \"kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.219569 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.219560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.219681 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.219601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320587 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320722 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320722 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcwt\" (UniqueName: \"kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320722 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320838 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.320886 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:01:38.320854    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:01:38.320922 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:01:38.320913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls podName:0bacc6e2-2c74-4d07-8919-9914625b945e nodeName:}" failed. No retries permitted until 2026-09-16 19:01:38.820893443 +0000 UTC m=+1282.539990513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:01:38.320993 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.320975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.321248 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.321227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.321362 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.321344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.329899 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.329871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcwt\" (UniqueName: \"kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.825614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.825582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.828045 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.828027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:38.994995 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:38.994956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:39.270514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:39.270487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:01:39.272647 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:01:39.272623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bacc6e2_2c74_4d07_8919_9914625b945e.slice/crio-ad2876fa626d67815fce383dd81911ef1e2876ef23c6275c4399431a8a41fff7 WatchSource:0}: Error finding container ad2876fa626d67815fce383dd81911ef1e2876ef23c6275c4399431a8a41fff7: Status 404 returned error can't find the container with id ad2876fa626d67815fce383dd81911ef1e2876ef23c6275c4399431a8a41fff7
Sep 16 19:01:39.551984 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:39.551958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerStarted","Data":"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"}
Sep 16 19:01:39.552169 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:39.552014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerStarted","Data":"ad2876fa626d67815fce383dd81911ef1e2876ef23c6275c4399431a8a41fff7"}
Sep 16 19:01:44.578354 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:44.578321    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerID="7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345" exitCode=0
Sep 16 19:01:44.578737 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:44.578395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerDied","Data":"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"}
Sep 16 19:01:45.586492 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:45.586435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerStarted","Data":"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"}
Sep 16 19:01:45.586890 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:45.586518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerStarted","Data":"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"}
Sep 16 19:01:45.586890 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:45.586734    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:45.586890 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:45.586799    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:01:45.628249 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:45.628200    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" podStartSLOduration=2.8298298920000002 podStartE2EDuration="7.628186361s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.798356469s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:45.626920957 +0000 UTC m=+1289.346018048" watchObservedRunningTime="2026-09-16 19:01:45.628186361 +0000 UTC m=+1289.347283438"
Sep 16 19:01:51.604707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:01:51.604676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:02:00.948531 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:00.948495    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:02:00.953440 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:00.953419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:00.956100 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:00.956079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 19:02:00.956202 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:00.956080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 19:02:00.962158 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:00.962127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:02:01.106197 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.106170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.106377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.106204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.106377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.106225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.106377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.106298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95w2f\" (UniqueName: \"kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.106377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.106369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.207745 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.207670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.207745 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.207711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.207946 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.207827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.207946 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.207874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95w2f\" (UniqueName: \"kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.207946 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.207924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.208118 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.208046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.208398 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.208376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.208507 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.208440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.210402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.210386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.218548 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.218524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95w2f\" (UniqueName: \"kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f\") pod \"isvc-sklearn-predictor-7698f4f677-f9tg2\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.264910 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.264884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:01.537279 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.537253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:02:01.539919 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:02:01.539877    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f1ef21_9b37_49df_839f_a547e51717a4.slice/crio-4d2fce5d66c23b0135d8e16c6aea493149f7598cf7a7c39a5d86bf5dccd9153c WatchSource:0}: Error finding container 4d2fce5d66c23b0135d8e16c6aea493149f7598cf7a7c39a5d86bf5dccd9153c: Status 404 returned error can't find the container with id 4d2fce5d66c23b0135d8e16c6aea493149f7598cf7a7c39a5d86bf5dccd9153c
Sep 16 19:02:01.662804 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.662768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerStarted","Data":"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"}
Sep 16 19:02:01.662997 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:01.662822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerStarted","Data":"4d2fce5d66c23b0135d8e16c6aea493149f7598cf7a7c39a5d86bf5dccd9153c"}
Sep 16 19:02:05.683168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:05.683133    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerID="d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d" exitCode=0
Sep 16 19:02:05.683575 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:05.683203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerDied","Data":"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"}
Sep 16 19:02:06.689717 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.689685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerStarted","Data":"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"}
Sep 16 19:02:06.690120 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.689745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerStarted","Data":"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"}
Sep 16 19:02:06.690120 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.690021    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:06.690212 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.690156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:06.691537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.691507    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:06.709428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:06.709386    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podStartSLOduration=3.055499903 podStartE2EDuration="6.709373134s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653873231s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:06.708390948 +0000 UTC m=+1310.427488047" watchObservedRunningTime="2026-09-16 19:02:06.709373134 +0000 UTC m=+1310.428470211"
Sep 16 19:02:07.695033 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:07.694996    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:12.699403 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:12.699375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:02:12.699904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:12.699877    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:22.700132 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:22.700094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:32.700256 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:32.700218    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:42.700004 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:42.699911    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:02:52.700353 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:02:52.700310    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:03:02.700421 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:02.700376    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:03:12.699858 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:12.699819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:03:14.744195 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:14.744149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:03:19.748224 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:19.748196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:03:24.749038 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:24.749013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:03:30.785682 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:30.785637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:03:31.065400 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:31.065368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:03:31.065724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:31.065673    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" containerID="cri-o://d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c" gracePeriod=30
Sep 16 19:03:31.065876 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:31.065718    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kube-rbac-proxy" containerID="cri-o://bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde" gracePeriod=30
Sep 16 19:03:32.080278 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:32.080246    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerID="bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde" exitCode=2
Sep 16 19:03:32.080656 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:32.080286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerDied","Data":"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"}
Sep 16 19:03:32.695875 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:32.695830    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 19:03:34.744941 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:34.744892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 19:03:34.936076 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:34.936040    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:03:34.936273 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:34.936253    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kserve-container" containerID="cri-o://b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c" gracePeriod=30
Sep 16 19:03:34.936365 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:34.936337    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kube-rbac-proxy" containerID="cri-o://352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1" gracePeriod=30
Sep 16 19:03:35.097949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.097914    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerID="352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1" exitCode=2
Sep 16 19:03:35.098097 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.097974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerDied","Data":"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"}
Sep 16 19:03:35.817715 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.817692    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:03:35.900983 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.900893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config\") pod \"c4f1ef21-9b37-49df-839f-a547e51717a4\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") "
Sep 16 19:03:35.900983 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.900939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95w2f\" (UniqueName: \"kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f\") pod \"c4f1ef21-9b37-49df-839f-a547e51717a4\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") "
Sep 16 19:03:35.900983 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.900983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls\") pod \"c4f1ef21-9b37-49df-839f-a547e51717a4\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") "
Sep 16 19:03:35.901275 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location\") pod \"c4f1ef21-9b37-49df-839f-a547e51717a4\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") "
Sep 16 19:03:35.901275 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert\") pod \"c4f1ef21-9b37-49df-839f-a547e51717a4\" (UID: \"c4f1ef21-9b37-49df-839f-a547e51717a4\") "
Sep 16 19:03:35.901478 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4f1ef21-9b37-49df-839f-a547e51717a4" (UID: "c4f1ef21-9b37-49df-839f-a547e51717a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:35.901478 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4f1ef21-9b37-49df-839f-a547e51717a4" (UID: "c4f1ef21-9b37-49df-839f-a547e51717a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:35.901588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:35.901588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901558    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f1ef21-9b37-49df-839f-a547e51717a4-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:35.901699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.901672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4f1ef21-9b37-49df-839f-a547e51717a4" (UID: "c4f1ef21-9b37-49df-839f-a547e51717a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:35.903347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.903315    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f" (OuterVolumeSpecName: "kube-api-access-95w2f") pod "c4f1ef21-9b37-49df-839f-a547e51717a4" (UID: "c4f1ef21-9b37-49df-839f-a547e51717a4"). InnerVolumeSpecName "kube-api-access-95w2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:35.903347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:35.903326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4f1ef21-9b37-49df-839f-a547e51717a4" (UID: "c4f1ef21-9b37-49df-839f-a547e51717a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:36.002640 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.002603    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95w2f\" (UniqueName: \"kubernetes.io/projected/c4f1ef21-9b37-49df-839f-a547e51717a4-kube-api-access-95w2f\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.002640 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.002631    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f1ef21-9b37-49df-839f-a547e51717a4-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.002640 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.002643    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f1ef21-9b37-49df-839f-a547e51717a4-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.106053 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.106023    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerID="d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c" exitCode=0
Sep 16 19:03:36.106246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.106097    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"
Sep 16 19:03:36.106246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.106119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerDied","Data":"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"}
Sep 16 19:03:36.106246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.106195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2" event={"ID":"c4f1ef21-9b37-49df-839f-a547e51717a4","Type":"ContainerDied","Data":"4d2fce5d66c23b0135d8e16c6aea493149f7598cf7a7c39a5d86bf5dccd9153c"}
Sep 16 19:03:36.106246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.106232    2579 scope.go:117] "RemoveContainer" containerID="bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"
Sep 16 19:03:36.116833 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.116818    2579 scope.go:117] "RemoveContainer" containerID="d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"
Sep 16 19:03:36.124883 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.124866    2579 scope.go:117] "RemoveContainer" containerID="d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"
Sep 16 19:03:36.132070 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.132051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:03:36.134047 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.134023    2579 scope.go:117] "RemoveContainer" containerID="bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"
Sep 16 19:03:36.134364 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:36.134341    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde\": container with ID starting with bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde not found: ID does not exist" containerID="bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"
Sep 16 19:03:36.134441 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.134373    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde"} err="failed to get container status \"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde\": rpc error: code = NotFound desc = could not find container \"bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde\": container with ID starting with bd1471c875ee4f3fe3bbc2b14f3941496ff81e4b506131691c8e7570864d3fde not found: ID does not exist"
Sep 16 19:03:36.134441 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.134393    2579 scope.go:117] "RemoveContainer" containerID="d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"
Sep 16 19:03:36.134699 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:36.134673    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c\": container with ID starting with d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c not found: ID does not exist" containerID="d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"
Sep 16 19:03:36.134754 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.134710    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c"} err="failed to get container status \"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c\": rpc error: code = NotFound desc = could not find container \"d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c\": container with ID starting with d52912de46dfcc530ff5c343939264b955c06946f7055998d1919e37e92da63c not found: ID does not exist"
Sep 16 19:03:36.134754 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.134735    2579 scope.go:117] "RemoveContainer" containerID="d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"
Sep 16 19:03:36.135167 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:36.135127    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d\": container with ID starting with d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d not found: ID does not exist" containerID="d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"
Sep 16 19:03:36.135270 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.135178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d"} err="failed to get container status \"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d\": rpc error: code = NotFound desc = could not find container \"d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d\": container with ID starting with d00100e388f77e0832b6cbd501c9b2a73f49d69378cb2783307091dad075e28d not found: ID does not exist"
Sep 16 19:03:36.138236 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.138025    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7698f4f677-f9tg2"]
Sep 16 19:03:36.376280 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.376258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:03:36.508026 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.507949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert\") pod \"0bacc6e2-2c74-4d07-8919-9914625b945e\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") "
Sep 16 19:03:36.508026 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.507982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwcwt\" (UniqueName: \"kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt\") pod \"0bacc6e2-2c74-4d07-8919-9914625b945e\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") "
Sep 16 19:03:36.508209 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.508037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config\") pod \"0bacc6e2-2c74-4d07-8919-9914625b945e\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") "
Sep 16 19:03:36.508209 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.508068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location\") pod \"0bacc6e2-2c74-4d07-8919-9914625b945e\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") "
Sep 16 19:03:36.508209 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.508104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") pod \"0bacc6e2-2c74-4d07-8919-9914625b945e\" (UID: \"0bacc6e2-2c74-4d07-8919-9914625b945e\") "
Sep 16 19:03:36.508429 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.508405    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bacc6e2-2c74-4d07-8919-9914625b945e" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:36.508519 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.508408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bacc6e2-2c74-4d07-8919-9914625b945e" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:36.510290 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.510264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bacc6e2-2c74-4d07-8919-9914625b945e" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:36.510435 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.510310    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt" (OuterVolumeSpecName: "kube-api-access-qwcwt") pod "0bacc6e2-2c74-4d07-8919-9914625b945e" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e"). InnerVolumeSpecName "kube-api-access-qwcwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:36.512579 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.512555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bacc6e2-2c74-4d07-8919-9914625b945e" (UID: "0bacc6e2-2c74-4d07-8919-9914625b945e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:36.609117 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.609081    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.609117 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.609110    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bacc6e2-2c74-4d07-8919-9914625b945e-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.609117 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.609120    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bacc6e2-2c74-4d07-8919-9914625b945e-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.609340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.609129    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bacc6e2-2c74-4d07-8919-9914625b945e-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.609340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.609137    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcwt\" (UniqueName: \"kubernetes.io/projected/0bacc6e2-2c74-4d07-8919-9914625b945e-kube-api-access-qwcwt\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:03:36.749882 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:36.749850    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" path="/var/lib/kubelet/pods/c4f1ef21-9b37-49df-839f-a547e51717a4/volumes"
Sep 16 19:03:37.111356 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.111320    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerID="b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c" exitCode=0
Sep 16 19:03:37.111795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.111381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"
Sep 16 19:03:37.111795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.111393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerDied","Data":"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"}
Sep 16 19:03:37.111795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.111450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8" event={"ID":"0bacc6e2-2c74-4d07-8919-9914625b945e","Type":"ContainerDied","Data":"ad2876fa626d67815fce383dd81911ef1e2876ef23c6275c4399431a8a41fff7"}
Sep 16 19:03:37.111795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.111502    2579 scope.go:117] "RemoveContainer" containerID="352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"
Sep 16 19:03:37.119754 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.119739    2579 scope.go:117] "RemoveContainer" containerID="b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"
Sep 16 19:03:37.127432 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.127416    2579 scope.go:117] "RemoveContainer" containerID="7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"
Sep 16 19:03:37.134061 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.134036    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:03:37.136725 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.136701    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-8llt8"]
Sep 16 19:03:37.137357 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.137341    2579 scope.go:117] "RemoveContainer" containerID="352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"
Sep 16 19:03:37.137636 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:37.137618    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1\": container with ID starting with 352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1 not found: ID does not exist" containerID="352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"
Sep 16 19:03:37.137713 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.137646    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1"} err="failed to get container status \"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1\": rpc error: code = NotFound desc = could not find container \"352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1\": container with ID starting with 352d17eff4d7924cf01701f2eab11dca84c2879b64b22f616352a9caef52a8d1 not found: ID does not exist"
Sep 16 19:03:37.137713 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.137669    2579 scope.go:117] "RemoveContainer" containerID="b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"
Sep 16 19:03:37.137944 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:37.137926    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c\": container with ID starting with b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c not found: ID does not exist" containerID="b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"
Sep 16 19:03:37.137983 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.137950    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c"} err="failed to get container status \"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c\": rpc error: code = NotFound desc = could not find container \"b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c\": container with ID starting with b34f8d07f942e17a3767a7811f7beb9db4a2949b06bdd38b58cda28a09cbb06c not found: ID does not exist"
Sep 16 19:03:37.137983 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.137964    2579 scope.go:117] "RemoveContainer" containerID="7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"
Sep 16 19:03:37.138194 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:03:37.138175    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345\": container with ID starting with 7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345 not found: ID does not exist" containerID="7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"
Sep 16 19:03:37.138255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:37.138201    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345"} err="failed to get container status \"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345\": rpc error: code = NotFound desc = could not find container \"7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345\": container with ID starting with 7f11cabd9701101eb7cd7fbf373e8d0e3c942835f808fd2a77179b934aa07345 not found: ID does not exist"
Sep 16 19:03:38.748375 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:38.748346    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" path="/var/lib/kubelet/pods/0bacc6e2-2c74-4d07-8919-9914625b945e/volumes"
Sep 16 19:03:40.151612 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151921    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151944    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="storage-initializer"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151950    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="storage-initializer"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151957    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="storage-initializer"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="storage-initializer"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151969    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kube-rbac-proxy"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151974    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kube-rbac-proxy"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151983    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kserve-container"
Sep 16 19:03:40.151982 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.151988    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kserve-container"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152002    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kube-rbac-proxy"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152008    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kube-rbac-proxy"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152063    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kserve-container"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kube-rbac-proxy"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152076    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bacc6e2-2c74-4d07-8919-9914625b945e" containerName="kube-rbac-proxy"
Sep 16 19:03:40.152294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.152081    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f1ef21-9b37-49df-839f-a547e51717a4" containerName="kserve-container"
Sep 16 19:03:40.157018 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.156998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.159663 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.159639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:03:40.159789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.159639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:40.159789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.159717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:40.159789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.159769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 19:03:40.159789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.159719    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:40.160622 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.160603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:03:40.160622 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.160613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:40.170016 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.169996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:03:40.240935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.240902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rrj\" (UniqueName: \"kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.241213 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.241190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.241336 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.241254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.241336 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.241290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.241336 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.241326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.342742 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.342713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.342867 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.342768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rrj\" (UniqueName: \"kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.342867 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.342798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.342867 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.342823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.342867 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.342843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.343272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.343248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.343447 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.343428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.343528 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.343499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.345540 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.345520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.353933 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.353911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rrj\" (UniqueName: \"kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.466750 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.466677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:03:40.741087 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:40.741060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:03:40.742891 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:03:40.742857    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144ec14_ab5b_421d_b6bb_3fe09f368d11.slice/crio-a8f5007518640da2f0fc89dc4d8630fa3979db426a98c3a6d9b77ff0d813f029 WatchSource:0}: Error finding container a8f5007518640da2f0fc89dc4d8630fa3979db426a98c3a6d9b77ff0d813f029: Status 404 returned error can't find the container with id a8f5007518640da2f0fc89dc4d8630fa3979db426a98c3a6d9b77ff0d813f029
Sep 16 19:03:41.132775 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:41.132737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerStarted","Data":"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"}
Sep 16 19:03:41.132977 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:03:41.132799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerStarted","Data":"a8f5007518640da2f0fc89dc4d8630fa3979db426a98c3a6d9b77ff0d813f029"}
Sep 16 19:04:10.266557 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:10.266524    2579 generic.go:356] "Generic (PLEG): container finished" podID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerID="4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7" exitCode=0
Sep 16 19:04:10.267092 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:10.266562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerDied","Data":"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"}
Sep 16 19:04:11.274658 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:11.274617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerStarted","Data":"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"}
Sep 16 19:04:11.275037 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:11.274700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerStarted","Data":"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"}
Sep 16 19:04:11.275037 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:11.275005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:04:11.294378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:11.294321    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" podStartSLOduration=2.822562699 podStartE2EDuration="31.294304702s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.471742003s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:11.293838242 +0000 UTC m=+1435.012935320" watchObservedRunningTime="2026-09-16 19:04:11.294304702 +0000 UTC m=+1435.013401779"
Sep 16 19:04:12.280417 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:12.280380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:04:18.290573 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:04:18.290541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:05:16.805781 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:16.805749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:05:16.809744 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:16.809723    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:05:41.747580 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:41.747508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:05:52.784611 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:52.784575    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:05:56.320492 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:56.320435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:05:56.320937 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:56.320741    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kserve-container" containerID="cri-o://92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d" gracePeriod=30
Sep 16 19:05:56.320937 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:56.320781    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kube-rbac-proxy" containerID="cri-o://0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953" gracePeriod=30
Sep 16 19:05:56.730985 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:56.730896    2579 generic.go:356] "Generic (PLEG): container finished" podID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerID="0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953" exitCode=2
Sep 16 19:05:56.730985 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:56.730968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerDied","Data":"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"}
Sep 16 19:05:57.662105 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.662081    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:05:57.737133 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.737057    2579 generic.go:356] "Generic (PLEG): container finished" podID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerID="92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d" exitCode=0
Sep 16 19:05:57.737133 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.737104    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"
Sep 16 19:05:57.737133 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.737127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerDied","Data":"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"}
Sep 16 19:05:57.737394 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.737222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn" event={"ID":"9144ec14-ab5b-421d-b6bb-3fe09f368d11","Type":"ContainerDied","Data":"a8f5007518640da2f0fc89dc4d8630fa3979db426a98c3a6d9b77ff0d813f029"}
Sep 16 19:05:57.737394 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.737262    2579 scope.go:117] "RemoveContainer" containerID="0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"
Sep 16 19:05:57.745610 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.745589    2579 scope.go:117] "RemoveContainer" containerID="92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"
Sep 16 19:05:57.753222 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.753206    2579 scope.go:117] "RemoveContainer" containerID="4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"
Sep 16 19:05:57.760527 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.760512    2579 scope.go:117] "RemoveContainer" containerID="0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"
Sep 16 19:05:57.760751 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:05:57.760734    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953\": container with ID starting with 0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953 not found: ID does not exist" containerID="0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"
Sep 16 19:05:57.760816 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.760759    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953"} err="failed to get container status \"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953\": rpc error: code = NotFound desc = could not find container \"0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953\": container with ID starting with 0f30c45eb04ec31b6266f273a2b5c1bbf2301a343b163c3a2324ede4cac69953 not found: ID does not exist"
Sep 16 19:05:57.760816 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.760777    2579 scope.go:117] "RemoveContainer" containerID="92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"
Sep 16 19:05:57.761018 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:05:57.761003    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d\": container with ID starting with 92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d not found: ID does not exist" containerID="92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"
Sep 16 19:05:57.761060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.761022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d"} err="failed to get container status \"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d\": rpc error: code = NotFound desc = could not find container \"92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d\": container with ID starting with 92a0757ac52b13876e539a5a69efbffb8f522ebc43c4f290573026ccc66a8c8d not found: ID does not exist"
Sep 16 19:05:57.761060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.761035    2579 scope.go:117] "RemoveContainer" containerID="4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"
Sep 16 19:05:57.761239 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:05:57.761224    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7\": container with ID starting with 4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7 not found: ID does not exist" containerID="4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"
Sep 16 19:05:57.761289 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.761242    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7"} err="failed to get container status \"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7\": rpc error: code = NotFound desc = could not find container \"4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7\": container with ID starting with 4e2464de0c0848fd920a8762d6bdc9c2c5937fea7ce76cd3114363e00339a4c7 not found: ID does not exist"
Sep 16 19:05:57.769531 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert\") pod \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") "
Sep 16 19:05:57.769604 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location\") pod \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") "
Sep 16 19:05:57.769649 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7rrj\" (UniqueName: \"kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj\") pod \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") "
Sep 16 19:05:57.769704 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config\") pod \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") "
Sep 16 19:05:57.769704 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls\") pod \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\" (UID: \"9144ec14-ab5b-421d-b6bb-3fe09f368d11\") "
Sep 16 19:05:57.769940 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.769914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9144ec14-ab5b-421d-b6bb-3fe09f368d11" (UID: "9144ec14-ab5b-421d-b6bb-3fe09f368d11"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:57.770057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.770016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9144ec14-ab5b-421d-b6bb-3fe09f368d11" (UID: "9144ec14-ab5b-421d-b6bb-3fe09f368d11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:57.771746 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.771722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj" (OuterVolumeSpecName: "kube-api-access-k7rrj") pod "9144ec14-ab5b-421d-b6bb-3fe09f368d11" (UID: "9144ec14-ab5b-421d-b6bb-3fe09f368d11"). InnerVolumeSpecName "kube-api-access-k7rrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:57.771827 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.771765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9144ec14-ab5b-421d-b6bb-3fe09f368d11" (UID: "9144ec14-ab5b-421d-b6bb-3fe09f368d11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:57.774192 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.774171    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9144ec14-ab5b-421d-b6bb-3fe09f368d11" (UID: "9144ec14-ab5b-421d-b6bb-3fe09f368d11"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:57.870433 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.870410    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:05:57.870433 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.870430    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rrj\" (UniqueName: \"kubernetes.io/projected/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-api-access-k7rrj\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:05:57.870599 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.870440    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:05:57.870599 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.870449    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9144ec14-ab5b-421d-b6bb-3fe09f368d11-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:05:57.870599 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:57.870470    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9144ec14-ab5b-421d-b6bb-3fe09f368d11-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:05:58.060397 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:58.060363    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:05:58.064749 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:58.064725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ljzgn"]
Sep 16 19:05:58.748017 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:05:58.747976    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" path="/var/lib/kubelet/pods/9144ec14-ab5b-421d-b6bb-3fe09f368d11/volumes"
Sep 16 19:07:01.572659 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572949    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kserve-container"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572959    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kserve-container"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kube-rbac-proxy"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572975    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kube-rbac-proxy"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572983    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="storage-initializer"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.572988    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="storage-initializer"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.573047    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kube-rbac-proxy"
Sep 16 19:07:01.573148 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.573055    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9144ec14-ab5b-421d-b6bb-3fe09f368d11" containerName="kserve-container"
Sep 16 19:07:01.576323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.576305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.579056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579029    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:07:01.579056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 19:07:01.579313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:07:01.579313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:07:01.579313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579140    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:07:01.579313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579157    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:07:01.579474 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.579431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:07:01.589689 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.589664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:01.692128 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.692092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.692128 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.692124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.692377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.692159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9s4j\" (UniqueName: \"kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.692377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.692178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.692377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.692247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9s4j\" (UniqueName: \"kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793309 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793309 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793309 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793635 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793853 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.793912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.793852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.795685 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.795667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.802684 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.802662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9s4j\" (UniqueName: \"kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j\") pod \"isvc-sklearn-runtime-predictor-b78d6df9c-vmwll\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:01.887313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:01.887207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:02.160877 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:02.160852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:02.163474 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:07:02.163433    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0dfc2e_9b82_48e6_a928_e82e2a1d007a.slice/crio-76192392a270f67021dacd1cc456b049bb4067182ac5a88879f9ede2b2cb5c35 WatchSource:0}: Error finding container 76192392a270f67021dacd1cc456b049bb4067182ac5a88879f9ede2b2cb5c35: Status 404 returned error can't find the container with id 76192392a270f67021dacd1cc456b049bb4067182ac5a88879f9ede2b2cb5c35
Sep 16 19:07:03.002271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:03.002235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerStarted","Data":"8f764bee45dba803c184395d4908d874badac5ab0471fc9ecbcb5f89d6618c4c"}
Sep 16 19:07:03.002682 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:03.002290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerStarted","Data":"76192392a270f67021dacd1cc456b049bb4067182ac5a88879f9ede2b2cb5c35"}
Sep 16 19:07:08.026547 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:08.026512    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerID="8f764bee45dba803c184395d4908d874badac5ab0471fc9ecbcb5f89d6618c4c" exitCode=0
Sep 16 19:07:08.026900 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:08.026578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerDied","Data":"8f764bee45dba803c184395d4908d874badac5ab0471fc9ecbcb5f89d6618c4c"}
Sep 16 19:07:09.034326 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.034290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerStarted","Data":"9f613133f67cac7af07eab2665926cb491de5712186d7dffc79754ad4d75a2f3"}
Sep 16 19:07:09.034741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.034349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerStarted","Data":"ece64a8074ae09452e66e20bf7269b60c8ed74dc2873349f2d7ccb598e87d968"}
Sep 16 19:07:09.034741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.034630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:09.034741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.034689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:09.035937 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.035912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:07:09.056477 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:09.056410    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podStartSLOduration=2.412416097 podStartE2EDuration="8.056396435s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.643980338s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:09.055318322 +0000 UTC m=+1612.774415423" watchObservedRunningTime="2026-09-16 19:07:09.056396435 +0000 UTC m=+1612.775493511"
Sep 16 19:07:10.039734 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:10.039695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:07:15.045028 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:15.044997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:15.045595 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:15.045566    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:07:25.046510 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:25.046474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:33.442227 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:33.442195    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:33.442751 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:33.442445    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container" containerID="cri-o://ece64a8074ae09452e66e20bf7269b60c8ed74dc2873349f2d7ccb598e87d968" gracePeriod=30
Sep 16 19:07:33.442751 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:33.442542    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kube-rbac-proxy" containerID="cri-o://9f613133f67cac7af07eab2665926cb491de5712186d7dffc79754ad4d75a2f3" gracePeriod=30
Sep 16 19:07:34.150621 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.150589    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerID="9f613133f67cac7af07eab2665926cb491de5712186d7dffc79754ad4d75a2f3" exitCode=2
Sep 16 19:07:34.150621 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.150614    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerID="ece64a8074ae09452e66e20bf7269b60c8ed74dc2873349f2d7ccb598e87d968" exitCode=0
Sep 16 19:07:34.150807 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.150635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerDied","Data":"9f613133f67cac7af07eab2665926cb491de5712186d7dffc79754ad4d75a2f3"}
Sep 16 19:07:34.150807 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.150681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerDied","Data":"ece64a8074ae09452e66e20bf7269b60c8ed74dc2873349f2d7ccb598e87d968"}
Sep 16 19:07:34.186783 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.186763    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:34.272350 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location\") pod \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") "
Sep 16 19:07:34.272502 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls\") pod \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") "
Sep 16 19:07:34.272502 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9s4j\" (UniqueName: \"kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j\") pod \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") "
Sep 16 19:07:34.272502 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config\") pod \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") "
Sep 16 19:07:34.272645 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert\") pod \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\" (UID: \"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a\") "
Sep 16 19:07:34.272841 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" (UID: "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:34.272913 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.272861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" (UID: "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:34.274512 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.274491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" (UID: "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:34.274703 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.274686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j" (OuterVolumeSpecName: "kube-api-access-k9s4j") pod "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" (UID: "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a"). InnerVolumeSpecName "kube-api-access-k9s4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:34.298656 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.298598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" (UID: "9d0dfc2e-9b82-48e6-a928-e82e2a1d007a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:34.373422 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.373402    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:07:34.373422 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.373421    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:07:34.373570 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.373433    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9s4j\" (UniqueName: \"kubernetes.io/projected/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-api-access-k9s4j\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:07:34.373570 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.373441    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:07:34.373570 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:34.373450    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:07:35.157195 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.157154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll" event={"ID":"9d0dfc2e-9b82-48e6-a928-e82e2a1d007a","Type":"ContainerDied","Data":"76192392a270f67021dacd1cc456b049bb4067182ac5a88879f9ede2b2cb5c35"}
Sep 16 19:07:35.157195 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.157179    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"
Sep 16 19:07:35.157798 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.157216    2579 scope.go:117] "RemoveContainer" containerID="9f613133f67cac7af07eab2665926cb491de5712186d7dffc79754ad4d75a2f3"
Sep 16 19:07:35.168644 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.168627    2579 scope.go:117] "RemoveContainer" containerID="ece64a8074ae09452e66e20bf7269b60c8ed74dc2873349f2d7ccb598e87d968"
Sep 16 19:07:35.176188 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.176052    2579 scope.go:117] "RemoveContainer" containerID="8f764bee45dba803c184395d4908d874badac5ab0471fc9ecbcb5f89d6618c4c"
Sep 16 19:07:35.178608 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.178589    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:35.183263 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:35.183242    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-b78d6df9c-vmwll"]
Sep 16 19:07:36.748712 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:07:36.748680    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" path="/var/lib/kubelet/pods/9d0dfc2e-9b82-48e6-a928-e82e2a1d007a/volumes"
Sep 16 19:09:13.896625 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.896579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897081    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="storage-initializer"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897101    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="storage-initializer"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897117    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897125    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897149    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kube-rbac-proxy"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897160    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kube-rbac-proxy"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897238    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kube-rbac-proxy"
Sep 16 19:09:13.897295 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.897256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0dfc2e-9b82-48e6-a928-e82e2a1d007a" containerName="kserve-container"
Sep 16 19:09:13.900416 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.900395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:13.903272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.903248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:09:13.903272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.903266    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:09:13.903448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.903274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:13.903448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.903312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:13.904496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.904475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 19:09:13.904614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.904494    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:13.904614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.904519    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:13.912727 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:13.912705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:09:14.066207 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.066160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.066372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.066228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.066372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.066272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.066372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.066296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.066372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.066311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhs2\" (UniqueName: \"kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.166748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.166678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.166748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.166724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.166902 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.166757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.166902 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.166780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.166902 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.166796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhs2\" (UniqueName: \"kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.167009 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:09:14.166919    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:09:14.167009 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:09:14.166998    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls podName:27b74d58-a84b-421a-bde2-cf872d11477c nodeName:}" failed. No retries permitted until 2026-09-16 19:09:14.666967362 +0000 UTC m=+1738.386064417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls") pod "isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:09:14.167125 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.167104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.167474 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.167436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.167535 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.167518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.178892 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.178871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhs2\" (UniqueName: \"kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.670727 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.670698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.673226 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.673202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5fd47fbf89-w5j95\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:14.811420 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:14.811388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:15.086731 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:15.086707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:09:15.088572 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:09:15.088539    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b74d58_a84b_421a_bde2_cf872d11477c.slice/crio-3f59fcaf5899809ffd81e136a29c9e6ad7ab5e54edd77febaf9774b6c872bf5a WatchSource:0}: Error finding container 3f59fcaf5899809ffd81e136a29c9e6ad7ab5e54edd77febaf9774b6c872bf5a: Status 404 returned error can't find the container with id 3f59fcaf5899809ffd81e136a29c9e6ad7ab5e54edd77febaf9774b6c872bf5a
Sep 16 19:09:15.587225 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:15.587182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerStarted","Data":"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"}
Sep 16 19:09:15.587430 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:15.587246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerStarted","Data":"3f59fcaf5899809ffd81e136a29c9e6ad7ab5e54edd77febaf9774b6c872bf5a"}
Sep 16 19:09:19.603964 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:19.603931    2579 generic.go:356] "Generic (PLEG): container finished" podID="27b74d58-a84b-421a-bde2-cf872d11477c" containerID="88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc" exitCode=0
Sep 16 19:09:19.604326 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:19.604004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerDied","Data":"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"}
Sep 16 19:09:20.611151 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:20.611116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerStarted","Data":"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"}
Sep 16 19:09:20.611553 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:20.611178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerStarted","Data":"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"}
Sep 16 19:09:20.611553 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:20.611423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:20.631472 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:20.631416    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podStartSLOduration=4.004512039 podStartE2EDuration="7.631392882s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626880843s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:20.629780439 +0000 UTC m=+1744.348877530" watchObservedRunningTime="2026-09-16 19:09:20.631392882 +0000 UTC m=+1744.350489959"
Sep 16 19:09:21.615368 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:21.615325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:21.616697 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:21.616669    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:22.620364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:22.620313    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:27.624913 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:27.624880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:09:27.625410 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:27.625379    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:35.461494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.461442    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:09:35.465156 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.465137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.467489 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.467441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:09:35.467610 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.467446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:09:35.476115 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.476090    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:09:35.501131 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.501109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:09:35.504489 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.504471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.506818 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.506802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:09:35.515536 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.515513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:09:35.542099 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.542188 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8q8\" (UniqueName: \"kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.542254 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.542315 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.542364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.542364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.542448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.542448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.542561 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.542561 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.542520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lf5\" (UniqueName: \"kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643110 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643274 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8q8\" (UniqueName: \"kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643274 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643274 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643274 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643274 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643574 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643574 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643574 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643574 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lf5\" (UniqueName: \"kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643764 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:09:35.643735    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:09:35.643821 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:09:35.643798    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls podName:df92fbe7-cd7e-40c9-b78f-a44257eb0b03 nodeName:}" failed. No retries permitted until 2026-09-16 19:09:36.14377978 +0000 UTC m=+1759.862876850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls") pod "isvc-canary-create-predictor-68cd88b8-zvdx9" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:09:35.643821 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.643935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.643935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.643910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.644447 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.644426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.644530 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.644480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.644676 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.644654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.646336 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.646302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.651866 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.651844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8q8\" (UniqueName: \"kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:35.652167 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.652143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lf5\" (UniqueName: \"kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5\") pod \"isvc-canary-create-v2-predictor-6c79c974bf-bdhcj\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:35.775863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:35.775790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:36.049236 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.049211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:09:36.051479 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:09:36.051433    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485ebf1_a534_41fa_949a_3a3a98898d20.slice/crio-0879fb2d833dc066e699984a814026bbcd181419245b9dbcc91de04af221c36b WatchSource:0}: Error finding container 0879fb2d833dc066e699984a814026bbcd181419245b9dbcc91de04af221c36b: Status 404 returned error can't find the container with id 0879fb2d833dc066e699984a814026bbcd181419245b9dbcc91de04af221c36b
Sep 16 19:09:36.148686 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.148647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:36.151149 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.151121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") pod \"isvc-canary-create-predictor-68cd88b8-zvdx9\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:36.415191 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.415162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:36.679925 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.679836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerStarted","Data":"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"}
Sep 16 19:09:36.679925 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.679890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerStarted","Data":"0879fb2d833dc066e699984a814026bbcd181419245b9dbcc91de04af221c36b"}
Sep 16 19:09:36.688518 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:36.688493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:09:36.690173 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:09:36.690151    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf92fbe7_cd7e_40c9_b78f_a44257eb0b03.slice/crio-381c02181b57e8de92c097b0ad19bd9aea31ea79571a3269922a6244428eb3c9 WatchSource:0}: Error finding container 381c02181b57e8de92c097b0ad19bd9aea31ea79571a3269922a6244428eb3c9: Status 404 returned error can't find the container with id 381c02181b57e8de92c097b0ad19bd9aea31ea79571a3269922a6244428eb3c9
Sep 16 19:09:37.625328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:37.625291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:37.686132 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:37.686093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerStarted","Data":"346ccf6b77dfa0a619e5cbd5fef52c91446fe5062188e0272e8f4c9a81bc71ce"}
Sep 16 19:09:37.686522 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:37.686189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerStarted","Data":"381c02181b57e8de92c097b0ad19bd9aea31ea79571a3269922a6244428eb3c9"}
Sep 16 19:09:40.702106 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:40.702077    2579 generic.go:356] "Generic (PLEG): container finished" podID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerID="8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef" exitCode=0
Sep 16 19:09:40.702479 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:40.702152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerDied","Data":"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"}
Sep 16 19:09:40.703901 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:40.703880    2579 generic.go:356] "Generic (PLEG): container finished" podID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerID="346ccf6b77dfa0a619e5cbd5fef52c91446fe5062188e0272e8f4c9a81bc71ce" exitCode=0
Sep 16 19:09:40.703998 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:40.703948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerDied","Data":"346ccf6b77dfa0a619e5cbd5fef52c91446fe5062188e0272e8f4c9a81bc71ce"}
Sep 16 19:09:41.710935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.710904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerStarted","Data":"34610086e8352b019d1920f338fe6a2e6a837c1194ecd23b7cf5a0ec89c40257"}
Sep 16 19:09:41.711370 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.710971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerStarted","Data":"c657b2757aa3c5b79748d24c8a658fc0bf28104a2d8b6e581ca64b131f8a755c"}
Sep 16 19:09:41.711370 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.711234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:41.715782 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.715752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerStarted","Data":"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"}
Sep 16 19:09:41.715929 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.715816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerStarted","Data":"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"}
Sep 16 19:09:41.716071 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.716052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:41.716119 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.716108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:41.717604 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.717578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:09:41.735007 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.734946    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podStartSLOduration=2.886833058 podStartE2EDuration="6.734928271s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848095213s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:41.731168156 +0000 UTC m=+1765.450265230" watchObservedRunningTime="2026-09-16 19:09:41.734928271 +0000 UTC m=+1765.454025349"
Sep 16 19:09:41.755634 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:41.755588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podStartSLOduration=2.987484804 podStartE2EDuration="6.755572527s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.768087723s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:41.753629398 +0000 UTC m=+1765.472726477" watchObservedRunningTime="2026-09-16 19:09:41.755572527 +0000 UTC m=+1765.474669604"
Sep 16 19:09:42.720536 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:42.720492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:09:42.720990 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:42.720721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:42.722002 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:42.721974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:09:43.726491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:43.726428    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:09:47.626175 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:47.626137    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:47.724814 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:47.724784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:09:47.725268 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:47.725242    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:09:48.730803 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:48.730778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:09:48.731283 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:48.731251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:09:57.625564 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:57.625520    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:09:57.725388 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:57.725348    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:09:58.731824 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:09:58.731788    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:07.626358 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:07.626277    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:10:07.725367 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:07.725325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:10:08.731537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:08.731495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:16.840166 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:16.840139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:10:16.845664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:16.845606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:10:17.625665 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:17.625624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:10:17.725872 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:17.725830    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:10:18.731504 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:18.731448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:27.626059 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:27.626019    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:10:27.725789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:27.725754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:10:28.731428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:28.731389    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:37.626632 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:37.626605    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:10:37.725399 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:37.725361    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:10:38.731672 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:38.731635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:44.023216 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:44.023181    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:10:44.023705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:44.023417    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" containerID="cri-o://bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608" gracePeriod=30
Sep 16 19:10:44.023705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:44.023483    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kube-rbac-proxy" containerID="cri-o://3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727" gracePeriod=30
Sep 16 19:10:45.023603 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:45.023570    2579 generic.go:356] "Generic (PLEG): container finished" podID="27b74d58-a84b-421a-bde2-cf872d11477c" containerID="3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727" exitCode=2
Sep 16 19:10:45.024039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:45.023629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerDied","Data":"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"}
Sep 16 19:10:47.620639 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:47.620597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:10:47.625941 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:47.625919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 16 19:10:47.725328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:47.725291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:10:48.466541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.466517    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:10:48.567827 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.567804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrhs2\" (UniqueName: \"kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2\") pod \"27b74d58-a84b-421a-bde2-cf872d11477c\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") "
Sep 16 19:10:48.567971 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.567839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config\") pod \"27b74d58-a84b-421a-bde2-cf872d11477c\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") "
Sep 16 19:10:48.567971 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.567885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location\") pod \"27b74d58-a84b-421a-bde2-cf872d11477c\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") "
Sep 16 19:10:48.567971 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.567905    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert\") pod \"27b74d58-a84b-421a-bde2-cf872d11477c\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") "
Sep 16 19:10:48.567971 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.567959    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") pod \"27b74d58-a84b-421a-bde2-cf872d11477c\" (UID: \"27b74d58-a84b-421a-bde2-cf872d11477c\") "
Sep 16 19:10:48.568259 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.568233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27b74d58-a84b-421a-bde2-cf872d11477c" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:48.568353 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.568299    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27b74d58-a84b-421a-bde2-cf872d11477c" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:48.568353 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.568325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27b74d58-a84b-421a-bde2-cf872d11477c" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:48.570099 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.570069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27b74d58-a84b-421a-bde2-cf872d11477c" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:48.570156 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.570119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2" (OuterVolumeSpecName: "kube-api-access-wrhs2") pod "27b74d58-a84b-421a-bde2-cf872d11477c" (UID: "27b74d58-a84b-421a-bde2-cf872d11477c"). InnerVolumeSpecName "kube-api-access-wrhs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:48.669625 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.669595    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27b74d58-a84b-421a-bde2-cf872d11477c-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.669625 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.669619    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhs2\" (UniqueName: \"kubernetes.io/projected/27b74d58-a84b-421a-bde2-cf872d11477c-kube-api-access-wrhs2\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.669625 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.669628    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.669996 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.669637    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27b74d58-a84b-421a-bde2-cf872d11477c-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.669996 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.669646    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27b74d58-a84b-421a-bde2-cf872d11477c-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:10:48.731811 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:48.731775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:49.045594 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.045506    2579 generic.go:356] "Generic (PLEG): container finished" podID="27b74d58-a84b-421a-bde2-cf872d11477c" containerID="bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608" exitCode=0
Sep 16 19:10:49.045594 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.045577    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"
Sep 16 19:10:49.045782 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.045613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerDied","Data":"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"}
Sep 16 19:10:49.045782 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.045700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95" event={"ID":"27b74d58-a84b-421a-bde2-cf872d11477c","Type":"ContainerDied","Data":"3f59fcaf5899809ffd81e136a29c9e6ad7ab5e54edd77febaf9774b6c872bf5a"}
Sep 16 19:10:49.045782 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.045741    2579 scope.go:117] "RemoveContainer" containerID="3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"
Sep 16 19:10:49.056025 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.056004    2579 scope.go:117] "RemoveContainer" containerID="bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"
Sep 16 19:10:49.064396 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.064377    2579 scope.go:117] "RemoveContainer" containerID="88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"
Sep 16 19:10:49.070929 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.070907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:10:49.073138 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073123    2579 scope.go:117] "RemoveContainer" containerID="3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"
Sep 16 19:10:49.073391 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:10:49.073373    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727\": container with ID starting with 3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727 not found: ID does not exist" containerID="3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"
Sep 16 19:10:49.073443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073401    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727"} err="failed to get container status \"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727\": rpc error: code = NotFound desc = could not find container \"3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727\": container with ID starting with 3c65d2671bb64a91ee1db0d9995f813b935f21ff7602c605a80d9ba427bb4727 not found: ID does not exist"
Sep 16 19:10:49.073443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073419    2579 scope.go:117] "RemoveContainer" containerID="bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"
Sep 16 19:10:49.073710 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:10:49.073693    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608\": container with ID starting with bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608 not found: ID does not exist" containerID="bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"
Sep 16 19:10:49.073751 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073718    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608"} err="failed to get container status \"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608\": rpc error: code = NotFound desc = could not find container \"bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608\": container with ID starting with bc6fdb1f0eb0f906df2aa23fbbdf71c8c727908cf9f7bc7b39025ac7b3e9e608 not found: ID does not exist"
Sep 16 19:10:49.073751 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073743    2579 scope.go:117] "RemoveContainer" containerID="88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"
Sep 16 19:10:49.073979 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:10:49.073964    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc\": container with ID starting with 88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc not found: ID does not exist" containerID="88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"
Sep 16 19:10:49.074015 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.073984    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc"} err="failed to get container status \"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc\": rpc error: code = NotFound desc = could not find container \"88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc\": container with ID starting with 88a7f34478bef7de04b916fe4af62612c7f2a82964a35a2c9cbcb432222b41fc not found: ID does not exist"
Sep 16 19:10:49.080048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:49.080029    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5fd47fbf89-w5j95"]
Sep 16 19:10:50.750562 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:50.750529    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" path="/var/lib/kubelet/pods/27b74d58-a84b-421a-bde2-cf872d11477c/volumes"
Sep 16 19:10:51.744307 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:51.744266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:10:57.726592 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:57.726561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:10:57.890138 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:57.890106    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:10:57.890358 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:57.890336    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container" containerID="cri-o://c657b2757aa3c5b79748d24c8a658fc0bf28104a2d8b6e581ca64b131f8a755c" gracePeriod=30
Sep 16 19:10:57.890444 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:57.890396    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kube-rbac-proxy" containerID="cri-o://34610086e8352b019d1920f338fe6a2e6a837c1194ecd23b7cf5a0ec89c40257" gracePeriod=30
Sep 16 19:10:57.891255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:57.891235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:10:58.089890 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:58.089842    2579 generic.go:356] "Generic (PLEG): container finished" podID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerID="34610086e8352b019d1920f338fe6a2e6a837c1194ecd23b7cf5a0ec89c40257" exitCode=2
Sep 16 19:10:58.090045 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:58.089913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerDied","Data":"34610086e8352b019d1920f338fe6a2e6a837c1194ecd23b7cf5a0ec89c40257"}
Sep 16 19:10:58.727316 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:10:58.727272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:11:02.111184 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.111150    2579 generic.go:356] "Generic (PLEG): container finished" podID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerID="c657b2757aa3c5b79748d24c8a658fc0bf28104a2d8b6e581ca64b131f8a755c" exitCode=0
Sep 16 19:11:02.111556 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.111226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerDied","Data":"c657b2757aa3c5b79748d24c8a658fc0bf28104a2d8b6e581ca64b131f8a755c"}
Sep 16 19:11:02.233208 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.233183    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:11:02.387784 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.387748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config\") pod \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") "
Sep 16 19:11:02.387942 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.387801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert\") pod \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") "
Sep 16 19:11:02.387942 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.387859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn8q8\" (UniqueName: \"kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8\") pod \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") "
Sep 16 19:11:02.387942 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.387912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location\") pod \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") "
Sep 16 19:11:02.388128 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388039    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") pod \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\" (UID: \"df92fbe7-cd7e-40c9-b78f-a44257eb0b03\") "
Sep 16 19:11:02.388211 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df92fbe7-cd7e-40c9-b78f-a44257eb0b03" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:02.388272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df92fbe7-cd7e-40c9-b78f-a44257eb0b03" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:02.388272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df92fbe7-cd7e-40c9-b78f-a44257eb0b03" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:02.388399 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388383    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:02.388477 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388406    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:02.388477 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.388422    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:02.390057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.390035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df92fbe7-cd7e-40c9-b78f-a44257eb0b03" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:02.390232 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.390211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8" (OuterVolumeSpecName: "kube-api-access-jn8q8") pod "df92fbe7-cd7e-40c9-b78f-a44257eb0b03" (UID: "df92fbe7-cd7e-40c9-b78f-a44257eb0b03"). InnerVolumeSpecName "kube-api-access-jn8q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:02.489405 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.489371    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:02.489405 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:02.489398    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn8q8\" (UniqueName: \"kubernetes.io/projected/df92fbe7-cd7e-40c9-b78f-a44257eb0b03-kube-api-access-jn8q8\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.117524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.117487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9" event={"ID":"df92fbe7-cd7e-40c9-b78f-a44257eb0b03","Type":"ContainerDied","Data":"381c02181b57e8de92c097b0ad19bd9aea31ea79571a3269922a6244428eb3c9"}
Sep 16 19:11:03.117924 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.117568    2579 scope.go:117] "RemoveContainer" containerID="34610086e8352b019d1920f338fe6a2e6a837c1194ecd23b7cf5a0ec89c40257"
Sep 16 19:11:03.117924 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.117503    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"
Sep 16 19:11:03.128624 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.128579    2579 scope.go:117] "RemoveContainer" containerID="c657b2757aa3c5b79748d24c8a658fc0bf28104a2d8b6e581ca64b131f8a755c"
Sep 16 19:11:03.136580 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.136553    2579 scope.go:117] "RemoveContainer" containerID="346ccf6b77dfa0a619e5cbd5fef52c91446fe5062188e0272e8f4c9a81bc71ce"
Sep 16 19:11:03.144921 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.144901    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:11:03.149621 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:03.149599    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68cd88b8-zvdx9"]
Sep 16 19:11:04.749314 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:04.749283    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" path="/var/lib/kubelet/pods/df92fbe7-cd7e-40c9-b78f-a44257eb0b03/volumes"
Sep 16 19:11:15.563525 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:15.563494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:11:15.563921 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:15.563732    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" containerID="cri-o://47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8" gracePeriod=30
Sep 16 19:11:15.563921 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:15.563754    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kube-rbac-proxy" containerID="cri-o://1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b" gracePeriod=30
Sep 16 19:11:16.180586 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:16.180554    2579 generic.go:356] "Generic (PLEG): container finished" podID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerID="1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b" exitCode=2
Sep 16 19:11:16.180759 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:16.180625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerDied","Data":"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"}
Sep 16 19:11:17.721724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:17.721628    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 16 19:11:17.726070 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:17.726042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:11:19.910273 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.910250    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:11:19.933591 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert\") pod \"f485ebf1-a534-41fa-949a-3a3a98898d20\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") "
Sep 16 19:11:19.933724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location\") pod \"f485ebf1-a534-41fa-949a-3a3a98898d20\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") "
Sep 16 19:11:19.933724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config\") pod \"f485ebf1-a534-41fa-949a-3a3a98898d20\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") "
Sep 16 19:11:19.933724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls\") pod \"f485ebf1-a534-41fa-949a-3a3a98898d20\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") "
Sep 16 19:11:19.933724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5lf5\" (UniqueName: \"kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5\") pod \"f485ebf1-a534-41fa-949a-3a3a98898d20\" (UID: \"f485ebf1-a534-41fa-949a-3a3a98898d20\") "
Sep 16 19:11:19.933954 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f485ebf1-a534-41fa-949a-3a3a98898d20" (UID: "f485ebf1-a534-41fa-949a-3a3a98898d20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:19.934016 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.933943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f485ebf1-a534-41fa-949a-3a3a98898d20" (UID: "f485ebf1-a534-41fa-949a-3a3a98898d20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:19.934070 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.934022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f485ebf1-a534-41fa-949a-3a3a98898d20" (UID: "f485ebf1-a534-41fa-949a-3a3a98898d20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:19.935981 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.935957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5" (OuterVolumeSpecName: "kube-api-access-g5lf5") pod "f485ebf1-a534-41fa-949a-3a3a98898d20" (UID: "f485ebf1-a534-41fa-949a-3a3a98898d20"). InnerVolumeSpecName "kube-api-access-g5lf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:19.936125 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:19.936098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f485ebf1-a534-41fa-949a-3a3a98898d20" (UID: "f485ebf1-a534-41fa-949a-3a3a98898d20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:20.034503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.034427    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f485ebf1-a534-41fa-949a-3a3a98898d20-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:20.034503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.034450    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5lf5\" (UniqueName: \"kubernetes.io/projected/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-api-access-g5lf5\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:20.034503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.034474    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:20.034503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.034483    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f485ebf1-a534-41fa-949a-3a3a98898d20-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:20.034503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.034491    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f485ebf1-a534-41fa-949a-3a3a98898d20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:11:20.200424 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.200392    2579 generic.go:356] "Generic (PLEG): container finished" podID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerID="47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8" exitCode=0
Sep 16 19:11:20.200601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.200448    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"
Sep 16 19:11:20.200601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.200474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerDied","Data":"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"}
Sep 16 19:11:20.200601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.200528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj" event={"ID":"f485ebf1-a534-41fa-949a-3a3a98898d20","Type":"ContainerDied","Data":"0879fb2d833dc066e699984a814026bbcd181419245b9dbcc91de04af221c36b"}
Sep 16 19:11:20.200601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.200555    2579 scope.go:117] "RemoveContainer" containerID="1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"
Sep 16 19:11:20.209423 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.209406    2579 scope.go:117] "RemoveContainer" containerID="47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"
Sep 16 19:11:20.216788 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.216770    2579 scope.go:117] "RemoveContainer" containerID="8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"
Sep 16 19:11:20.224298 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.224260    2579 scope.go:117] "RemoveContainer" containerID="1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"
Sep 16 19:11:20.224964 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:11:20.224939    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b\": container with ID starting with 1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b not found: ID does not exist" containerID="1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"
Sep 16 19:11:20.225060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.224976    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b"} err="failed to get container status \"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b\": rpc error: code = NotFound desc = could not find container \"1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b\": container with ID starting with 1839baa5d771a4b7502f2ea64958c029335c3a37745aa14f952ae95c0659171b not found: ID does not exist"
Sep 16 19:11:20.225060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.225002    2579 scope.go:117] "RemoveContainer" containerID="47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"
Sep 16 19:11:20.225307 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:11:20.225286    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8\": container with ID starting with 47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8 not found: ID does not exist" containerID="47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"
Sep 16 19:11:20.225399 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.225314    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8"} err="failed to get container status \"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8\": rpc error: code = NotFound desc = could not find container \"47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8\": container with ID starting with 47feabd4b43b73c69f0dc9fd858fae7d29233e34c9116c6d4a61c48ea7d09de8 not found: ID does not exist"
Sep 16 19:11:20.225399 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.225329    2579 scope.go:117] "RemoveContainer" containerID="8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"
Sep 16 19:11:20.225719 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:11:20.225699    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef\": container with ID starting with 8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef not found: ID does not exist" containerID="8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"
Sep 16 19:11:20.225790 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.225727    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef"} err="failed to get container status \"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef\": rpc error: code = NotFound desc = could not find container \"8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef\": container with ID starting with 8b563130ec1cb7b0314b5a2babcf643c58abcf4277ac57eaa4ee4a22f1a243ef not found: ID does not exist"
Sep 16 19:11:20.226730 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.226711    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:11:20.231873 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.231853    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6c79c974bf-bdhcj"]
Sep 16 19:11:20.749698 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.749661    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" path="/var/lib/kubelet/pods/f485ebf1-a534-41fa-949a-3a3a98898d20/volumes"
Sep 16 19:11:20.771996 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.771963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:11:20.772275 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772263    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772277    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772285    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772290    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772305    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772313    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772318    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772333    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772341    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772355    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772372    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772380    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772397    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772406    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772419    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="storage-initializer"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772544    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772553    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kube-rbac-proxy"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772560    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f485ebf1-a534-41fa-949a-3a3a98898d20" containerName="kserve-container"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772568    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df92fbe7-cd7e-40c9-b78f-a44257eb0b03" containerName="kserve-container"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772576    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kserve-container"
Sep 16 19:11:20.772588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.772582    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b74d58-a84b-421a-bde2-cf872d11477c" containerName="kube-rbac-proxy"
Sep 16 19:11:20.776934 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.776914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.779427 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.779406    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:20.779716 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.779698    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:11:20.779828 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.779737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:20.779828 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.779741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:20.779828 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.779776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:11:20.780611 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.780595    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:20.780659 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.780619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:11:20.787795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.787773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:11:20.833951 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.833920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:11:20.837390 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.837371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.839680 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.839664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:11:20.841258 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.841235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8wf\" (UniqueName: \"kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.841360 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.841277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.841360 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.841298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.841360 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.841324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.841535 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.841384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.849651 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.849631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:11:20.942620 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwqsp\" (UniqueName: \"kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8wf\" (UniqueName: \"kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.942842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:20.943373 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.943171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943518 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.943501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.943646 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.943628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.945439 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.945423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:20.951612 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:20.951591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8wf\" (UniqueName: \"kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf\") pod \"isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:21.043356 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwqsp\" (UniqueName: \"kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.043356 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.043595 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.043595 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.043595 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.043799 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.043776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.044089 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.044066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.044203 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.044182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.046056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.046016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.054818 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.054794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwqsp\" (UniqueName: \"kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp\") pod \"isvc-canary-promote-predictor-674955ff8d-ns5mk\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.087879 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.087856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:21.147216 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.147168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:21.365609 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.365578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:11:21.367282 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:11:21.367253    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c62b2e_f698_42ce_a336_da034e525a8e.slice/crio-59a77cc2e80419a1c0edf355fb45a85fa4077aa5c75e0a50d3f0ae4ed5420beb WatchSource:0}: Error finding container 59a77cc2e80419a1c0edf355fb45a85fa4077aa5c75e0a50d3f0ae4ed5420beb: Status 404 returned error can't find the container with id 59a77cc2e80419a1c0edf355fb45a85fa4077aa5c75e0a50d3f0ae4ed5420beb
Sep 16 19:11:21.427139 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:21.427116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:11:21.428614 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:11:21.428591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ef1e12_ecf7_463c_a914_e51d6b31705f.slice/crio-0449f81178e6ff5b624a15750553d9772db9e53acf689fed7fe0ee43dfae2ef6 WatchSource:0}: Error finding container 0449f81178e6ff5b624a15750553d9772db9e53acf689fed7fe0ee43dfae2ef6: Status 404 returned error can't find the container with id 0449f81178e6ff5b624a15750553d9772db9e53acf689fed7fe0ee43dfae2ef6
Sep 16 19:11:22.214906 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:22.214857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerStarted","Data":"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"}
Sep 16 19:11:22.215486 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:22.214966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerStarted","Data":"0449f81178e6ff5b624a15750553d9772db9e53acf689fed7fe0ee43dfae2ef6"}
Sep 16 19:11:22.216589 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:22.216565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerStarted","Data":"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"}
Sep 16 19:11:22.216687 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:22.216613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerStarted","Data":"59a77cc2e80419a1c0edf355fb45a85fa4077aa5c75e0a50d3f0ae4ed5420beb"}
Sep 16 19:11:25.233093 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:25.233059    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerID="24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442" exitCode=0
Sep 16 19:11:25.233551 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:25.233140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerDied","Data":"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"}
Sep 16 19:11:26.240812 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.240780    2579 generic.go:356] "Generic (PLEG): container finished" podID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerID="fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384" exitCode=0
Sep 16 19:11:26.241220 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.240847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerDied","Data":"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"}
Sep 16 19:11:26.243110 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.243090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerStarted","Data":"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"}
Sep 16 19:11:26.243219 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.243164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerStarted","Data":"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"}
Sep 16 19:11:26.243446 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.243429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:26.304096 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:26.303995    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podStartSLOduration=2.717123458 podStartE2EDuration="6.303974645s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586851187s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:26.303220523 +0000 UTC m=+1870.022317611" watchObservedRunningTime="2026-09-16 19:11:26.303974645 +0000 UTC m=+1870.023071735"
Sep 16 19:11:27.250014 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.249971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerStarted","Data":"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"}
Sep 16 19:11:27.250437 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.250284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:27.250437 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.250337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerStarted","Data":"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"}
Sep 16 19:11:27.250585 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.250486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:27.251705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.251677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:11:27.272614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:27.272565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podStartSLOduration=3.3280265509999998 podStartE2EDuration="7.27255181s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.944525259s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:27.271589301 +0000 UTC m=+1870.990686378" watchObservedRunningTime="2026-09-16 19:11:27.27255181 +0000 UTC m=+1870.991648887"
Sep 16 19:11:28.254894 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:28.254852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:11:28.255323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:28.255029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:28.256183 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:28.256158    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:29.260280 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:29.260236    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:33.259047 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:33.259018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:11:33.261444 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:33.259642    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:11:34.265049 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:34.265023    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:11:34.265561 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:34.265534    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:43.260108 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:43.260024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:11:44.265896 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:44.265855    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:53.260191 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:53.260150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:11:54.266104 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:11:54.266067    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:03.259601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:03.259560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:12:04.266314 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:04.266277    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:13.259584 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:13.259544    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:12:14.266047 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:14.266007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:23.260160 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:23.260122    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:12:24.266251 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:24.266212    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:33.260395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:33.260357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:12:34.266286 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.266249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:34.695085 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.695051    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:12:34.698807 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.698791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.701705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.701681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 19:12:34.701835 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.701732    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 19:12:34.714866 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.714844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:12:34.757103 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.757069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7c28\" (UniqueName: \"kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.757253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.757109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.757253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.757125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.757253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.757143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.757253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.757241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.857750 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.857718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.857904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.857795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7c28\" (UniqueName: \"kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.857904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.857830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.857904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.857845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.857904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.857860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.858270 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.858248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.858539 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.858512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.858539 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.858519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.860582 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.860553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:34.872915 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:34.872895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7c28\" (UniqueName: \"kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28\") pod \"isvc-tensorflow-predictor-547dfc5669-stxg2\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:35.009369 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:35.009279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:35.299545 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:35.299481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:12:35.302172 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:12:35.302143    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37a9e20_acc2_4c66_a799_58271843fc0a.slice/crio-ea37fc8821227e0301e23364c3412fbee5029869158eddf54994b9f6fcc25fde WatchSource:0}: Error finding container ea37fc8821227e0301e23364c3412fbee5029869158eddf54994b9f6fcc25fde: Status 404 returned error can't find the container with id ea37fc8821227e0301e23364c3412fbee5029869158eddf54994b9f6fcc25fde
Sep 16 19:12:35.568606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:35.568559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerStarted","Data":"e718dc27dab9893703e2e861e26072b5d4f2f50e2ebffaac4f058db9146a7a35"}
Sep 16 19:12:35.568798 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:35.568642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerStarted","Data":"ea37fc8821227e0301e23364c3412fbee5029869158eddf54994b9f6fcc25fde"}
Sep 16 19:12:35.745003 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:35.744955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:12:40.593925 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:40.593892    2579 generic.go:356] "Generic (PLEG): container finished" podID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerID="e718dc27dab9893703e2e861e26072b5d4f2f50e2ebffaac4f058db9146a7a35" exitCode=0
Sep 16 19:12:40.594388 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:40.593976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerDied","Data":"e718dc27dab9893703e2e861e26072b5d4f2f50e2ebffaac4f058db9146a7a35"}
Sep 16 19:12:40.595340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:40.595326    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:12:43.260192 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:43.260162    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:12:43.415187 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:43.415143    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:12:43.415683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:43.415444    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container" containerID="cri-o://c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994" gracePeriod=30
Sep 16 19:12:43.415683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:43.415559    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kube-rbac-proxy" containerID="cri-o://991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145" gracePeriod=30
Sep 16 19:12:43.416515 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:43.416483    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:12:44.260602 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.260551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:12:44.628264 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.628237    2579 generic.go:356] "Generic (PLEG): container finished" podID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerID="991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145" exitCode=2
Sep 16 19:12:44.628377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.628310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerDied","Data":"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"}
Sep 16 19:12:44.630368 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.630346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerStarted","Data":"485df53651c609d434272415dffd83ec8332694903319c3a8093a61628a461bd"}
Sep 16 19:12:44.630476 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.630401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerStarted","Data":"72a4f464d63f8e88220b6e6783ed1667b75af36ba9a2e33d61727038e659b4c6"}
Sep 16 19:12:44.630664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.630646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:44.654570 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:44.654530    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podStartSLOduration=2.228123012 podStartE2EDuration="10.654511277s" totalImagesPullingTime="3.854711859s" totalInitContainerRuntime="4.571676406s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:44.652773722 +0000 UTC m=+1948.371870800" watchObservedRunningTime="2026-09-16 19:12:44.654511277 +0000 UTC m=+1948.373608354"
Sep 16 19:12:45.634799 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:45.634763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:45.636119 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:45.636089    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:12:46.639515 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:46.639441    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:12:48.165805 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.165783    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:12:48.275477 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config\") pod \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") "
Sep 16 19:12:48.275477 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location\") pod \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") "
Sep 16 19:12:48.275699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwqsp\" (UniqueName: \"kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp\") pod \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") "
Sep 16 19:12:48.275699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert\") pod \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") "
Sep 16 19:12:48.275699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls\") pod \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\" (UID: \"a7ef1e12-ecf7-463c-a914-e51d6b31705f\") "
Sep 16 19:12:48.275849 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7ef1e12-ecf7-463c-a914-e51d6b31705f" (UID: "a7ef1e12-ecf7-463c-a914-e51d6b31705f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:48.275849 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7ef1e12-ecf7-463c-a914-e51d6b31705f" (UID: "a7ef1e12-ecf7-463c-a914-e51d6b31705f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:48.275919 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.275866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7ef1e12-ecf7-463c-a914-e51d6b31705f" (UID: "a7ef1e12-ecf7-463c-a914-e51d6b31705f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:48.277746 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.277719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp" (OuterVolumeSpecName: "kube-api-access-fwqsp") pod "a7ef1e12-ecf7-463c-a914-e51d6b31705f" (UID: "a7ef1e12-ecf7-463c-a914-e51d6b31705f"). InnerVolumeSpecName "kube-api-access-fwqsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:48.277864 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.277785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7ef1e12-ecf7-463c-a914-e51d6b31705f" (UID: "a7ef1e12-ecf7-463c-a914-e51d6b31705f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:48.376949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.376919    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwqsp\" (UniqueName: \"kubernetes.io/projected/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-api-access-fwqsp\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:12:48.376949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.376944    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:12:48.376949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.376954    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7ef1e12-ecf7-463c-a914-e51d6b31705f-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:12:48.377147 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.376962    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:12:48.377147 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.376971    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7ef1e12-ecf7-463c-a914-e51d6b31705f-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:12:48.650591 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.650557    2579 generic.go:356] "Generic (PLEG): container finished" podID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerID="c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994" exitCode=0
Sep 16 19:12:48.650770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.650615    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"
Sep 16 19:12:48.650770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.650615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerDied","Data":"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"}
Sep 16 19:12:48.650770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.650686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk" event={"ID":"a7ef1e12-ecf7-463c-a914-e51d6b31705f","Type":"ContainerDied","Data":"0449f81178e6ff5b624a15750553d9772db9e53acf689fed7fe0ee43dfae2ef6"}
Sep 16 19:12:48.650770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.650725    2579 scope.go:117] "RemoveContainer" containerID="991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"
Sep 16 19:12:48.659693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.659677    2579 scope.go:117] "RemoveContainer" containerID="c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"
Sep 16 19:12:48.667496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.667447    2579 scope.go:117] "RemoveContainer" containerID="fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"
Sep 16 19:12:48.674486 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.674470    2579 scope.go:117] "RemoveContainer" containerID="991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"
Sep 16 19:12:48.674752 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:12:48.674730    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145\": container with ID starting with 991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145 not found: ID does not exist" containerID="991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"
Sep 16 19:12:48.674797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.674767    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145"} err="failed to get container status \"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145\": rpc error: code = NotFound desc = could not find container \"991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145\": container with ID starting with 991d73ea87068a31c344d05f64a6ac0e9290aecfab7e9072a883cf2f69ab2145 not found: ID does not exist"
Sep 16 19:12:48.674797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.674785    2579 scope.go:117] "RemoveContainer" containerID="c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"
Sep 16 19:12:48.675033 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:12:48.675014    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994\": container with ID starting with c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994 not found: ID does not exist" containerID="c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"
Sep 16 19:12:48.675098 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.675043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994"} err="failed to get container status \"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994\": rpc error: code = NotFound desc = could not find container \"c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994\": container with ID starting with c5ac64b892baf2820543b8f4ebc1e316f0c08aac02280f1c1ea4b834b0821994 not found: ID does not exist"
Sep 16 19:12:48.675098 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.675066    2579 scope.go:117] "RemoveContainer" containerID="fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"
Sep 16 19:12:48.675315 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:12:48.675297    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384\": container with ID starting with fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384 not found: ID does not exist" containerID="fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"
Sep 16 19:12:48.675354 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.675329    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384"} err="failed to get container status \"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384\": rpc error: code = NotFound desc = could not find container \"fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384\": container with ID starting with fbc9f11f430abf6168b9ab4349a5f0e7d7d1553f81e8590e4121473e3455e384 not found: ID does not exist"
Sep 16 19:12:48.681815 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.681792    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:12:48.686084 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.686065    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-674955ff8d-ns5mk"]
Sep 16 19:12:48.749780 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:48.749748    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" path="/var/lib/kubelet/pods/a7ef1e12-ecf7-463c-a914-e51d6b31705f/volumes"
Sep 16 19:12:51.643928 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:51.643898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:12:51.644335 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:12:51.644311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:13:01.644823 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:01.644794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:13:14.772606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:14.772530    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:13:14.772963 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:14.772758    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container" containerID="cri-o://72a4f464d63f8e88220b6e6783ed1667b75af36ba9a2e33d61727038e659b4c6" gracePeriod=30
Sep 16 19:13:14.772963 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:14.772795    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" containerID="cri-o://485df53651c609d434272415dffd83ec8332694903319c3a8093a61628a461bd" gracePeriod=30
Sep 16 19:13:15.778939 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:15.778908    2579 generic.go:356] "Generic (PLEG): container finished" podID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerID="485df53651c609d434272415dffd83ec8332694903319c3a8093a61628a461bd" exitCode=2
Sep 16 19:13:15.779301 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:15.778977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerDied","Data":"485df53651c609d434272415dffd83ec8332694903319c3a8093a61628a461bd"}
Sep 16 19:13:16.640078 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:16.640036    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:21.640091 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:21.640050    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:26.640632 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:26.640588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:26.641011 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:26.640714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:13:31.639783 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:31.639741    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:36.639706 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:36.639662    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:41.640133 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:41.640082    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:13:44.910848 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:44.910811    2579 generic.go:356] "Generic (PLEG): container finished" podID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerID="72a4f464d63f8e88220b6e6783ed1667b75af36ba9a2e33d61727038e659b4c6" exitCode=137
Sep 16 19:13:44.911150 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:44.910886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerDied","Data":"72a4f464d63f8e88220b6e6783ed1667b75af36ba9a2e33d61727038e659b4c6"}
Sep 16 19:13:45.419940 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.419910    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:13:45.537213 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7c28\" (UniqueName: \"kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28\") pod \"b37a9e20-acc2-4c66-a799-58271843fc0a\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") "
Sep 16 19:13:45.537213 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls\") pod \"b37a9e20-acc2-4c66-a799-58271843fc0a\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") "
Sep 16 19:13:45.537494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location\") pod \"b37a9e20-acc2-4c66-a799-58271843fc0a\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") "
Sep 16 19:13:45.537494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert\") pod \"b37a9e20-acc2-4c66-a799-58271843fc0a\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") "
Sep 16 19:13:45.537494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config\") pod \"b37a9e20-acc2-4c66-a799-58271843fc0a\" (UID: \"b37a9e20-acc2-4c66-a799-58271843fc0a\") "
Sep 16 19:13:45.537702 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b37a9e20-acc2-4c66-a799-58271843fc0a" (UID: "b37a9e20-acc2-4c66-a799-58271843fc0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:45.537764 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.537716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b37a9e20-acc2-4c66-a799-58271843fc0a" (UID: "b37a9e20-acc2-4c66-a799-58271843fc0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:45.539588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.539527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b37a9e20-acc2-4c66-a799-58271843fc0a" (UID: "b37a9e20-acc2-4c66-a799-58271843fc0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:45.540152 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.540125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28" (OuterVolumeSpecName: "kube-api-access-n7c28") pod "b37a9e20-acc2-4c66-a799-58271843fc0a" (UID: "b37a9e20-acc2-4c66-a799-58271843fc0a"). InnerVolumeSpecName "kube-api-access-n7c28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:45.546580 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.546556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b37a9e20-acc2-4c66-a799-58271843fc0a" (UID: "b37a9e20-acc2-4c66-a799-58271843fc0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:45.638076 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.638052    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7c28\" (UniqueName: \"kubernetes.io/projected/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-api-access-n7c28\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.638076 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.638075    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b37a9e20-acc2-4c66-a799-58271843fc0a-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.638224 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.638085    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b37a9e20-acc2-4c66-a799-58271843fc0a-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.638224 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.638093    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.638224 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.638101    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b37a9e20-acc2-4c66-a799-58271843fc0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.917394 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.917363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2" event={"ID":"b37a9e20-acc2-4c66-a799-58271843fc0a","Type":"ContainerDied","Data":"ea37fc8821227e0301e23364c3412fbee5029869158eddf54994b9f6fcc25fde"}
Sep 16 19:13:45.917813 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.917408    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"
Sep 16 19:13:45.917813 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.917422    2579 scope.go:117] "RemoveContainer" containerID="485df53651c609d434272415dffd83ec8332694903319c3a8093a61628a461bd"
Sep 16 19:13:45.926708 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.926692    2579 scope.go:117] "RemoveContainer" containerID="72a4f464d63f8e88220b6e6783ed1667b75af36ba9a2e33d61727038e659b4c6"
Sep 16 19:13:45.934134 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.934113    2579 scope.go:117] "RemoveContainer" containerID="e718dc27dab9893703e2e861e26072b5d4f2f50e2ebffaac4f058db9146a7a35"
Sep 16 19:13:45.945214 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.945193    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:13:45.954339 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:45.954317    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-stxg2"]
Sep 16 19:13:46.749883 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:13:46.749849    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" path="/var/lib/kubelet/pods/b37a9e20-acc2-4c66-a799-58271843fc0a/volumes"
Sep 16 19:14:07.747016 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.746976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747356    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="storage-initializer"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747370    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="storage-initializer"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747379    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="storage-initializer"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747385    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="storage-initializer"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747404    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kube-rbac-proxy"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747410    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kube-rbac-proxy"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747417    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747423    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747440    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747446    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747451    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747472    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747530    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747538    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ef1e12-ecf7-463c-a914-e51d6b31705f" containerName="kube-rbac-proxy"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747545    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kserve-container"
Sep 16 19:14:07.747563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.747551    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37a9e20-acc2-4c66-a799-58271843fc0a" containerName="kube-rbac-proxy"
Sep 16 19:14:07.750608 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.750592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:07.753913 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.753892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 16 19:14:07.754111 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.754090    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 16 19:14:07.762333 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.762313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:14:07.928218 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.928176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:07.928218 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.928215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54xq\" (UniqueName: \"kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:07.928443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.928234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:07.928443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.928372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:07.928443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:07.928433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.029856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.029769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.029856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.029813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.029856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.029834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.030132 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.029858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54xq\" (UniqueName: \"kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.030132 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.029880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.030364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.030344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.030583 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.030563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.030753 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.030580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.032537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.032517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.039217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.039198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54xq\" (UniqueName: \"kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq\") pod \"isvc-triton-predictor-5d7db58676-xl8qh\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.061837 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.061810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:14:08.345247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:08.345222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:14:08.347848 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:14:08.347813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e173739_8fd3_484a_8678_fd0acd136222.slice/crio-c022202791a3fc8c47f17c76feb99cf599dafd02968ba5eaf90f0ee3182d455a WatchSource:0}: Error finding container c022202791a3fc8c47f17c76feb99cf599dafd02968ba5eaf90f0ee3182d455a: Status 404 returned error can't find the container with id c022202791a3fc8c47f17c76feb99cf599dafd02968ba5eaf90f0ee3182d455a
Sep 16 19:14:09.028192 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:09.028158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerStarted","Data":"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"}
Sep 16 19:14:09.028714 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:09.028215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerStarted","Data":"c022202791a3fc8c47f17c76feb99cf599dafd02968ba5eaf90f0ee3182d455a"}
Sep 16 19:14:13.048019 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:13.047984    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e173739-8fd3-484a-8678-fd0acd136222" containerID="6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43" exitCode=0
Sep 16 19:14:13.048404 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:13.048028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerDied","Data":"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"}
Sep 16 19:14:41.097820 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:41.097597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:14:41.098377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:41.097901    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" containerID="cri-o://181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585" gracePeriod=30
Sep 16 19:14:41.098377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:41.098025    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kube-rbac-proxy" containerID="cri-o://5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3" gracePeriod=30
Sep 16 19:14:41.248030 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:41.247992    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerID="5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3" exitCode=2
Sep 16 19:14:41.248208 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:41.248069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerDied","Data":"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"}
Sep 16 19:14:43.255766 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:43.255719    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 16 19:14:43.260368 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:43.260326    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:46.485243 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.484379    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:14:46.500861 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.500816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.507614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.507558    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:14:46.507718 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.507673    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:14:46.507765 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.507750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:14:46.594308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.594231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.594308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.594310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbbg\" (UniqueName: \"kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.594597 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.594348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.594597 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.594386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.594597 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.594443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.695558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.695655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.695690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbbg\" (UniqueName: \"kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:14:46.695835    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:14:46.695913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls podName:d1be0ea3-7711-4f92-9c36-32967d4b61da nodeName:}" failed. No retries permitted until 2026-09-16 19:14:47.195896966 +0000 UTC m=+2070.914994032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls") pod "isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.695934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.695968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696810 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.696372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696810 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.696480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.696810 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.696772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.709156 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.709130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbbg\" (UniqueName: \"kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:46.739438 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.739410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:14:46.898111 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf8wf\" (UniqueName: \"kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf\") pod \"b6c62b2e-f698-42ce-a336-da034e525a8e\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") "
Sep 16 19:14:46.898308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert\") pod \"b6c62b2e-f698-42ce-a336-da034e525a8e\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") "
Sep 16 19:14:46.898308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location\") pod \"b6c62b2e-f698-42ce-a336-da034e525a8e\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") "
Sep 16 19:14:46.898308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls\") pod \"b6c62b2e-f698-42ce-a336-da034e525a8e\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") "
Sep 16 19:14:46.898308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config\") pod \"b6c62b2e-f698-42ce-a336-da034e525a8e\" (UID: \"b6c62b2e-f698-42ce-a336-da034e525a8e\") "
Sep 16 19:14:46.898646 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898599    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6c62b2e-f698-42ce-a336-da034e525a8e" (UID: "b6c62b2e-f698-42ce-a336-da034e525a8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:46.898646 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6c62b2e-f698-42ce-a336-da034e525a8e" (UID: "b6c62b2e-f698-42ce-a336-da034e525a8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.898910 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.898861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6c62b2e-f698-42ce-a336-da034e525a8e" (UID: "b6c62b2e-f698-42ce-a336-da034e525a8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:46.901114 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.901078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6c62b2e-f698-42ce-a336-da034e525a8e" (UID: "b6c62b2e-f698-42ce-a336-da034e525a8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:46.901753 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.901717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf" (OuterVolumeSpecName: "kube-api-access-cf8wf") pod "b6c62b2e-f698-42ce-a336-da034e525a8e" (UID: "b6c62b2e-f698-42ce-a336-da034e525a8e"). InnerVolumeSpecName "kube-api-access-cf8wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:47.000039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.999914    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c62b2e-f698-42ce-a336-da034e525a8e-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.000039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.999955    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6c62b2e-f698-42ce-a336-da034e525a8e-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.000039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.999972    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.000039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:46.999986    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf8wf\" (UniqueName: \"kubernetes.io/projected/b6c62b2e-f698-42ce-a336-da034e525a8e-kube-api-access-cf8wf\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.000039 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.000001    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c62b2e-f698-42ce-a336-da034e525a8e-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:14:47.202514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.202322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:47.205471 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.205426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") pod \"isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:47.290052 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.289962    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerID="181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585" exitCode=0
Sep 16 19:14:47.290052 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.290044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerDied","Data":"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"}
Sep 16 19:14:47.290291 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.290109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw" event={"ID":"b6c62b2e-f698-42ce-a336-da034e525a8e","Type":"ContainerDied","Data":"59a77cc2e80419a1c0edf355fb45a85fa4077aa5c75e0a50d3f0ae4ed5420beb"}
Sep 16 19:14:47.290291 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.290148    2579 scope.go:117] "RemoveContainer" containerID="5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"
Sep 16 19:14:47.290416 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.290316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"
Sep 16 19:14:47.304177 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.304155    2579 scope.go:117] "RemoveContainer" containerID="181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"
Sep 16 19:14:47.314981 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.314959    2579 scope.go:117] "RemoveContainer" containerID="24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"
Sep 16 19:14:47.325563 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.325525    2579 scope.go:117] "RemoveContainer" containerID="5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"
Sep 16 19:14:47.325880 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:14:47.325853    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3\": container with ID starting with 5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3 not found: ID does not exist" containerID="5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"
Sep 16 19:14:47.325968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.325892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3"} err="failed to get container status \"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3\": rpc error: code = NotFound desc = could not find container \"5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3\": container with ID starting with 5445b8d1f9e25d1e27cfaaf29be986c33ed8683219625653ff84845ecc6bd3f3 not found: ID does not exist"
Sep 16 19:14:47.325968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.325918    2579 scope.go:117] "RemoveContainer" containerID="181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"
Sep 16 19:14:47.326216 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:14:47.326186    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585\": container with ID starting with 181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585 not found: ID does not exist" containerID="181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"
Sep 16 19:14:47.326323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.326225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585"} err="failed to get container status \"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585\": rpc error: code = NotFound desc = could not find container \"181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585\": container with ID starting with 181bcbf92f654f3887184d3655b85882147830f533ecaeb1bb1aa6d947ff0585 not found: ID does not exist"
Sep 16 19:14:47.326323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.326249    2579 scope.go:117] "RemoveContainer" containerID="24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"
Sep 16 19:14:47.326832 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:14:47.326790    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442\": container with ID starting with 24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442 not found: ID does not exist" containerID="24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"
Sep 16 19:14:47.326915 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.326826    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442"} err="failed to get container status \"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442\": rpc error: code = NotFound desc = could not find container \"24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442\": container with ID starting with 24c736454511d4ea5f44dcca253feeb4297b36e345dbc8a15112e25c28d17442 not found: ID does not exist"
Sep 16 19:14:47.328048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.328007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:14:47.338752 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.338699    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-85cf669bc9-xf9tw"]
Sep 16 19:14:47.420405 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.420364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:47.744786 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:47.744553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:14:47.746582 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:14:47.746544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1be0ea3_7711_4f92_9c36_32967d4b61da.slice/crio-333d666880ca11823d5558706a9d869a8f852b507098e9421c57ce0ccb264672 WatchSource:0}: Error finding container 333d666880ca11823d5558706a9d869a8f852b507098e9421c57ce0ccb264672: Status 404 returned error can't find the container with id 333d666880ca11823d5558706a9d869a8f852b507098e9421c57ce0ccb264672
Sep 16 19:14:48.302718 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:48.302523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerStarted","Data":"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"}
Sep 16 19:14:48.302718 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:48.302595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerStarted","Data":"333d666880ca11823d5558706a9d869a8f852b507098e9421c57ce0ccb264672"}
Sep 16 19:14:48.769089 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:48.768999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" path="/var/lib/kubelet/pods/b6c62b2e-f698-42ce-a336-da034e525a8e/volumes"
Sep 16 19:14:52.333324 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:52.333288    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerID="3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7" exitCode=0
Sep 16 19:14:52.333912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:52.333370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerDied","Data":"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"}
Sep 16 19:14:53.342707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.342576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerStarted","Data":"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"}
Sep 16 19:14:53.342707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.342656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerStarted","Data":"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"}
Sep 16 19:14:53.343230 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.342966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:53.343286 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.343216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:53.344729 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.344430    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:14:53.371165 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:53.371102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podStartSLOduration=3.084181666 podStartE2EDuration="7.371083593s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.286901927s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:53.369386564 +0000 UTC m=+2077.088483653" watchObservedRunningTime="2026-09-16 19:14:53.371083593 +0000 UTC m=+2077.090180671"
Sep 16 19:14:54.351651 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:54.351331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:14:59.356508 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:59.356388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:14:59.357100 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:14:59.357065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:09.357689 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:09.357642    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:19.357627 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:19.357575    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:29.357164 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:29.357100    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:39.357899 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:39.357858    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:49.358168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:49.358114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:15:59.357964 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:15:59.357917    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:07.619215 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:07.619145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:16:07.619685 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:07.619280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:16:08.752919 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.752829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerStarted","Data":"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"}
Sep 16 19:16:08.752919 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.752879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerStarted","Data":"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"}
Sep 16 19:16:08.753358 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.753147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:08.753358 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.753294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:08.754448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.754417    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:08.778526 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:08.774970    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podStartSLOduration=2.393441894 podStartE2EDuration="2m1.774949969s" totalImagesPullingTime="1m55.30387302s" totalInitContainerRuntime="4.077635055s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:08.774113872 +0000 UTC m=+2152.493210960" watchObservedRunningTime="2026-09-16 19:16:08.774949969 +0000 UTC m=+2152.494047048"
Sep 16 19:16:09.357900 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:09.357863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:09.757931 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:09.757846    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:14.762734 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:14.762710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:14.763216 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:14.763159    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:15.744028 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:15.743976    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:24.763854 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:24.763821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:25.744600 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:25.744574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:16:30.097540 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:30.097504    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:16:30.097936 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:30.097742    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container" containerID="cri-o://6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb" gracePeriod=30
Sep 16 19:16:30.097936 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:30.097795    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kube-rbac-proxy" containerID="cri-o://055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332" gracePeriod=30
Sep 16 19:16:30.856365 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:30.856331    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e173739-8fd3-484a-8678-fd0acd136222" containerID="055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332" exitCode=2
Sep 16 19:16:30.856544 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:30.856397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerDied","Data":"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"}
Sep 16 19:16:32.267589 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.267566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:32.452881 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.452804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls\") pod \"0e173739-8fd3-484a-8678-fd0acd136222\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") "
Sep 16 19:16:32.452881 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.452836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert\") pod \"0e173739-8fd3-484a-8678-fd0acd136222\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") "
Sep 16 19:16:32.452881 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.452859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location\") pod \"0e173739-8fd3-484a-8678-fd0acd136222\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") "
Sep 16 19:16:32.453168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.452882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config\") pod \"0e173739-8fd3-484a-8678-fd0acd136222\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") "
Sep 16 19:16:32.453168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.452982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j54xq\" (UniqueName: \"kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq\") pod \"0e173739-8fd3-484a-8678-fd0acd136222\" (UID: \"0e173739-8fd3-484a-8678-fd0acd136222\") "
Sep 16 19:16:32.453282 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.453240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e173739-8fd3-484a-8678-fd0acd136222" (UID: "0e173739-8fd3-484a-8678-fd0acd136222"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:32.453331 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.453278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e173739-8fd3-484a-8678-fd0acd136222" (UID: "0e173739-8fd3-484a-8678-fd0acd136222"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:32.453402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.453354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e173739-8fd3-484a-8678-fd0acd136222" (UID: "0e173739-8fd3-484a-8678-fd0acd136222"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:32.455102 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.455084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e173739-8fd3-484a-8678-fd0acd136222" (UID: "0e173739-8fd3-484a-8678-fd0acd136222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:32.455257 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.455242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq" (OuterVolumeSpecName: "kube-api-access-j54xq") pod "0e173739-8fd3-484a-8678-fd0acd136222" (UID: "0e173739-8fd3-484a-8678-fd0acd136222"). InnerVolumeSpecName "kube-api-access-j54xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:32.554127 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.554106    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e173739-8fd3-484a-8678-fd0acd136222-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:16:32.554127 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.554130    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:16:32.554251 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.554139    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e173739-8fd3-484a-8678-fd0acd136222-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:16:32.554251 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.554148    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e173739-8fd3-484a-8678-fd0acd136222-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:16:32.554251 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.554156    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j54xq\" (UniqueName: \"kubernetes.io/projected/0e173739-8fd3-484a-8678-fd0acd136222-kube-api-access-j54xq\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:16:32.866830 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.866793    2579 generic.go:356] "Generic (PLEG): container finished" podID="0e173739-8fd3-484a-8678-fd0acd136222" containerID="6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb" exitCode=0
Sep 16 19:16:32.867024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.866852    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"
Sep 16 19:16:32.867024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.866854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerDied","Data":"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"}
Sep 16 19:16:32.867024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.866966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh" event={"ID":"0e173739-8fd3-484a-8678-fd0acd136222","Type":"ContainerDied","Data":"c022202791a3fc8c47f17c76feb99cf599dafd02968ba5eaf90f0ee3182d455a"}
Sep 16 19:16:32.867024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.866991    2579 scope.go:117] "RemoveContainer" containerID="055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"
Sep 16 19:16:32.875445 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.875426    2579 scope.go:117] "RemoveContainer" containerID="6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"
Sep 16 19:16:32.882986 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.882969    2579 scope.go:117] "RemoveContainer" containerID="6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"
Sep 16 19:16:32.890340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.890321    2579 scope.go:117] "RemoveContainer" containerID="055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"
Sep 16 19:16:32.890640 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:16:32.890612    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332\": container with ID starting with 055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332 not found: ID does not exist" containerID="055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"
Sep 16 19:16:32.890724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.890651    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332"} err="failed to get container status \"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332\": rpc error: code = NotFound desc = could not find container \"055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332\": container with ID starting with 055f37ff9442fb0b4033ce849902dc609b7c4175cd1ef901e978415e84815332 not found: ID does not exist"
Sep 16 19:16:32.890724 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.890676    2579 scope.go:117] "RemoveContainer" containerID="6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"
Sep 16 19:16:32.890934 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:16:32.890918    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb\": container with ID starting with 6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb not found: ID does not exist" containerID="6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"
Sep 16 19:16:32.890989 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.890939    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb"} err="failed to get container status \"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb\": rpc error: code = NotFound desc = could not find container \"6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb\": container with ID starting with 6fc63375694ee69d8f7462c081c781703233a6f7005d20dfa4ad5da402ab9ccb not found: ID does not exist"
Sep 16 19:16:32.890989 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.890954    2579 scope.go:117] "RemoveContainer" containerID="6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"
Sep 16 19:16:32.891226 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:16:32.891210    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43\": container with ID starting with 6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43 not found: ID does not exist" containerID="6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"
Sep 16 19:16:32.891266 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.891232    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43"} err="failed to get container status \"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43\": rpc error: code = NotFound desc = could not find container \"6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43\": container with ID starting with 6cbad84fd8eac0c4ad9005d3628bb12e56232c4396f4bf89e58146a0fe51bf43 not found: ID does not exist"
Sep 16 19:16:32.896879 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.896857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:16:32.903495 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:32.903473    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xl8qh"]
Sep 16 19:16:34.749097 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:34.749066    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e173739-8fd3-484a-8678-fd0acd136222" path="/var/lib/kubelet/pods/0e173739-8fd3-484a-8678-fd0acd136222/volumes"
Sep 16 19:16:35.272726 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.272694    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:16:35.273056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container"
Sep 16 19:16:35.273056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273068    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="storage-initializer"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273074    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="storage-initializer"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273087    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273093    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273099    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273112    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273127    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="storage-initializer"
Sep 16 19:16:35.273155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273133    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="storage-initializer"
Sep 16 19:16:35.273555 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kserve-container"
Sep 16 19:16:35.273555 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273196    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e173739-8fd3-484a-8678-fd0acd136222" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273555 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273203    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kube-rbac-proxy"
Sep 16 19:16:35.273555 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.273211    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c62b2e-f698-42ce-a336-da034e525a8e" containerName="kserve-container"
Sep 16 19:16:35.286775 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.286741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.286928 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.286891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:16:35.289566 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.289544    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 16 19:16:35.289683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.289624    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 19:16:35.479991 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.479958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.480168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.480038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.480168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.480066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.480168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.480092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494jm\" (UniqueName: \"kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.480301 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.480190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581439 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581664 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494jm\" (UniqueName: \"kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.581895 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:16:35.581681    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 16 19:16:35.581895 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:16:35.581796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls podName:d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:36.08177922 +0000 UTC m=+2179.800876279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-km76b" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 16 19:16:35.582017 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.581945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.582243 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.582223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.582322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.582261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:35.591202 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:35.591180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494jm\" (UniqueName: \"kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:36.086255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.086222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:36.088830 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.088802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-km76b\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:36.197434 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.197399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:16:36.493179 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.493003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:16:36.496034 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:16:36.495998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86fa6c7_5256_46d9_bbed_f7eb9b70a6a0.slice/crio-3ed6090f35cceb5c072c770ca275d4ec92f90102158e95b00d60472fa891d180 WatchSource:0}: Error finding container 3ed6090f35cceb5c072c770ca275d4ec92f90102158e95b00d60472fa891d180: Status 404 returned error can't find the container with id 3ed6090f35cceb5c072c770ca275d4ec92f90102158e95b00d60472fa891d180
Sep 16 19:16:36.887363 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.887321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerStarted","Data":"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"}
Sep 16 19:16:36.887549 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:36.887381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerStarted","Data":"3ed6090f35cceb5c072c770ca275d4ec92f90102158e95b00d60472fa891d180"}
Sep 16 19:16:40.912534 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:40.912501    2579 generic.go:356] "Generic (PLEG): container finished" podID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerID="35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92" exitCode=0
Sep 16 19:16:40.912869 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:16:40.912574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerDied","Data":"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"}
Sep 16 19:17:01.016945 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:01.016857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerStarted","Data":"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"}
Sep 16 19:17:02.023348 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:02.023318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerStarted","Data":"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"}
Sep 16 19:17:02.023838 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:02.023540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:17:02.023838 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:02.023705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:17:02.025004 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:02.024981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:02.047273 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:02.047224    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podStartSLOduration=2.851141632 podStartE2EDuration="27.047209842s" totalImagesPullingTime="20.031990658s" totalInitContainerRuntime="4.164077552s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:02.045517214 +0000 UTC m=+2205.764614293" watchObservedRunningTime="2026-09-16 19:17:02.047209842 +0000 UTC m=+2205.766306920"
Sep 16 19:17:03.027791 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:03.027751    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:08.032756 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:08.032724    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:17:08.033305 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:08.033276    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:18.033413 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:18.033372    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:28.033491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:28.033430    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:38.033314 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:38.033227    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:46.503284 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:46.503252    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:17:46.503778 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:46.503555    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container" containerID="cri-o://2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446" gracePeriod=30
Sep 16 19:17:46.503778 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:46.503647    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kube-rbac-proxy" containerID="cri-o://1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6" gracePeriod=30
Sep 16 19:17:47.226924 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:47.226892    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerID="1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6" exitCode=2
Sep 16 19:17:47.227100 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:47.226937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerDied","Data":"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"}
Sep 16 19:17:48.034010 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:48.033969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:49.351606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:49.351564    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 16 19:17:50.850917 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.850893    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:17:50.940658 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.940580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") pod \"d1be0ea3-7711-4f92-9c36-32967d4b61da\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") "
Sep 16 19:17:50.940658 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.940619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config\") pod \"d1be0ea3-7711-4f92-9c36-32967d4b61da\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") "
Sep 16 19:17:50.940854 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.940678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lbbg\" (UniqueName: \"kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg\") pod \"d1be0ea3-7711-4f92-9c36-32967d4b61da\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") "
Sep 16 19:17:50.940854 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.940703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location\") pod \"d1be0ea3-7711-4f92-9c36-32967d4b61da\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") "
Sep 16 19:17:50.940854 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.940746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert\") pod \"d1be0ea3-7711-4f92-9c36-32967d4b61da\" (UID: \"d1be0ea3-7711-4f92-9c36-32967d4b61da\") "
Sep 16 19:17:50.941044 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.941012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1be0ea3-7711-4f92-9c36-32967d4b61da" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:50.941111 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.941092    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1be0ea3-7711-4f92-9c36-32967d4b61da" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.941172 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.941107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1be0ea3-7711-4f92-9c36-32967d4b61da" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.943048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.943027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1be0ea3-7711-4f92-9c36-32967d4b61da" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:50.943161 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:50.943142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg" (OuterVolumeSpecName: "kube-api-access-5lbbg") pod "d1be0ea3-7711-4f92-9c36-32967d4b61da" (UID: "d1be0ea3-7711-4f92-9c36-32967d4b61da"). InnerVolumeSpecName "kube-api-access-5lbbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:51.041968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.041927    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lbbg\" (UniqueName: \"kubernetes.io/projected/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-api-access-5lbbg\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.041968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.041958    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1be0ea3-7711-4f92-9c36-32967d4b61da-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.041968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.041970    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.041968 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.041979    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1be0ea3-7711-4f92-9c36-32967d4b61da-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.042234 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.041987    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1be0ea3-7711-4f92-9c36-32967d4b61da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.247395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.247302    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerID="2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446" exitCode=0
Sep 16 19:17:51.247395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.247371    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"
Sep 16 19:17:51.247395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.247383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerDied","Data":"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"}
Sep 16 19:17:51.247705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.247443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t" event={"ID":"d1be0ea3-7711-4f92-9c36-32967d4b61da","Type":"ContainerDied","Data":"333d666880ca11823d5558706a9d869a8f852b507098e9421c57ce0ccb264672"}
Sep 16 19:17:51.247705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.247489    2579 scope.go:117] "RemoveContainer" containerID="1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"
Sep 16 19:17:51.256371 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.256356    2579 scope.go:117] "RemoveContainer" containerID="2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"
Sep 16 19:17:51.264073 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.264054    2579 scope.go:117] "RemoveContainer" containerID="3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"
Sep 16 19:17:51.271444 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.271429    2579 scope.go:117] "RemoveContainer" containerID="1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"
Sep 16 19:17:51.271746 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:17:51.271727    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6\": container with ID starting with 1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6 not found: ID does not exist" containerID="1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"
Sep 16 19:17:51.271809 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.271755    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6"} err="failed to get container status \"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6\": rpc error: code = NotFound desc = could not find container \"1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6\": container with ID starting with 1b4aca8e14909b3b39c924342d698057c4a73e8ddd65061b851813e5b14df9e6 not found: ID does not exist"
Sep 16 19:17:51.271809 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.271771    2579 scope.go:117] "RemoveContainer" containerID="2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"
Sep 16 19:17:51.272046 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:17:51.272027    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446\": container with ID starting with 2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446 not found: ID does not exist" containerID="2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"
Sep 16 19:17:51.272084 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.272054    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446"} err="failed to get container status \"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446\": rpc error: code = NotFound desc = could not find container \"2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446\": container with ID starting with 2310cf803c533a35cdca011d3672504a66cf8f8811abbee379a9a6a67ba47446 not found: ID does not exist"
Sep 16 19:17:51.272084 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.272069    2579 scope.go:117] "RemoveContainer" containerID="3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"
Sep 16 19:17:51.272306 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:17:51.272286    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7\": container with ID starting with 3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7 not found: ID does not exist" containerID="3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"
Sep 16 19:17:51.272364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.272316    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7"} err="failed to get container status \"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7\": rpc error: code = NotFound desc = could not find container \"3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7\": container with ID starting with 3d88263a47706a3f17f1f021650d4d9e9e96189a9c6b2f8b2f14707d5a419ca7 not found: ID does not exist"
Sep 16 19:17:51.280546 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.280524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:17:51.289514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.289496    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6bd4b7c9bd-vnj8t"]
Sep 16 19:17:51.728065 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728034    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:17:51.728432 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728418    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="storage-initializer"
Sep 16 19:17:51.728514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728435    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="storage-initializer"
Sep 16 19:17:51.728514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728468    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container"
Sep 16 19:17:51.728514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container"
Sep 16 19:17:51.728514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728492    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kube-rbac-proxy"
Sep 16 19:17:51.728514 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728498    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kube-rbac-proxy"
Sep 16 19:17:51.728673 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728554    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kserve-container"
Sep 16 19:17:51.728673 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.728563    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" containerName="kube-rbac-proxy"
Sep 16 19:17:51.730559 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.730544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.733428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.733406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 19:17:51.733881 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.733864    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:17:51.747009 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.746987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:17:51.836985 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.836952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:17:51.839313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.839299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.842129 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.842100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:17:51.849742 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.849488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.849742 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.849587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.849742 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.849694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.849742 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.849740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6g8\" (UniqueName: \"kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.850032 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.849764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.851734 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.851713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:17:51.951127 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.951305 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.951305 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.951305 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.951541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6g8\" (UniqueName: \"kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.951541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.951541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.951541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.951541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncz6\" (UniqueName: \"kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.951772 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:51.951772 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.951720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.952334 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.952304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.952621 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.952602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.953838 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.953818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:51.960640 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:51.960617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6g8\" (UniqueName: \"kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8\") pod \"isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:52.040959 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.040898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:52.052642 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.052615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.052887 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.052670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.052887 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.052722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.052887 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.052747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncz6\" (UniqueName: \"kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.052887 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.052770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.052887 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:17:52.052862    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:17:52.053157 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:17:52.052944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls podName:c2565e05-8d64-4e00-b0c7-473368ca93c4 nodeName:}" failed. No retries permitted until 2026-09-16 19:17:52.552923595 +0000 UTC m=+2256.272020665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls") pod "isvc-canary-stop-predictor-69f76fb68d-4dh7t" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:17:52.053157 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.053059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.053449 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.053428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.053574 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.053512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.066088 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.066053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncz6\" (UniqueName: \"kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.317266 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.317227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:17:52.319692 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:17:52.319662    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79513fcc_6d6d_4280_8385_c5aae49d8f72.slice/crio-d7ffef9c4eec2cd7a8b2319ecc00b1ef3a6401792abfb1c1ea14b9cb7d0a3bd8 WatchSource:0}: Error finding container d7ffef9c4eec2cd7a8b2319ecc00b1ef3a6401792abfb1c1ea14b9cb7d0a3bd8: Status 404 returned error can't find the container with id d7ffef9c4eec2cd7a8b2319ecc00b1ef3a6401792abfb1c1ea14b9cb7d0a3bd8
Sep 16 19:17:52.558562 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.558532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.561019 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.560999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") pod \"isvc-canary-stop-predictor-69f76fb68d-4dh7t\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:52.750194 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.750119    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1be0ea3-7711-4f92-9c36-32967d4b61da" path="/var/lib/kubelet/pods/d1be0ea3-7711-4f92-9c36-32967d4b61da/volumes"
Sep 16 19:17:52.750500 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:52.750469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:53.035537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:53.033843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:17:53.041707 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:17:53.041681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2565e05_8d64_4e00_b0c7_473368ca93c4.slice/crio-1c97d29f3c601501dc3ec47f9e32413b498e1522f0d5f2d96c2bada523b4d5ec WatchSource:0}: Error finding container 1c97d29f3c601501dc3ec47f9e32413b498e1522f0d5f2d96c2bada523b4d5ec: Status 404 returned error can't find the container with id 1c97d29f3c601501dc3ec47f9e32413b498e1522f0d5f2d96c2bada523b4d5ec
Sep 16 19:17:53.260161 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:53.260118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerStarted","Data":"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"}
Sep 16 19:17:53.260354 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:53.260199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerStarted","Data":"1c97d29f3c601501dc3ec47f9e32413b498e1522f0d5f2d96c2bada523b4d5ec"}
Sep 16 19:17:53.261699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:53.261674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerStarted","Data":"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"}
Sep 16 19:17:53.261799 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:53.261724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerStarted","Data":"d7ffef9c4eec2cd7a8b2319ecc00b1ef3a6401792abfb1c1ea14b9cb7d0a3bd8"}
Sep 16 19:17:57.281345 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:57.281310    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerID="1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66" exitCode=0
Sep 16 19:17:57.281800 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:57.281383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerDied","Data":"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"}
Sep 16 19:17:57.283200 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:57.283180    2579 generic.go:356] "Generic (PLEG): container finished" podID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerID="f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593" exitCode=0
Sep 16 19:17:57.283329 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:57.283209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerDied","Data":"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"}
Sep 16 19:17:58.033343 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.033307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:17:58.290272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.290186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerStarted","Data":"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"}
Sep 16 19:17:58.290272 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.290259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerStarted","Data":"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"}
Sep 16 19:17:58.290743 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.290473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:58.292293 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.292270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerStarted","Data":"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"}
Sep 16 19:17:58.292428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.292321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerStarted","Data":"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"}
Sep 16 19:17:58.292606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.292590    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:58.315765 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.315718    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podStartSLOduration=3.287965256 podStartE2EDuration="7.315700824s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.027735568s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:58.31357945 +0000 UTC m=+2262.032676527" watchObservedRunningTime="2026-09-16 19:17:58.315700824 +0000 UTC m=+2262.034797902"
Sep 16 19:17:58.334140 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:58.334089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podStartSLOduration=3.425532183 podStartE2EDuration="7.334071689s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908539506s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:58.332969894 +0000 UTC m=+2262.052066971" watchObservedRunningTime="2026-09-16 19:17:58.334071689 +0000 UTC m=+2262.053168767"
Sep 16 19:17:59.297155 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:59.297121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:17:59.297591 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:59.297179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:17:59.298217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:59.298188    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:17:59.298310 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:17:59.298207    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:00.301479 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:00.301416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:00.301884 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:00.301550    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:05.306500 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:05.306448    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:18:05.307056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:05.306969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:18:05.307056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:05.307023    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:05.307384 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:05.307363    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:08.034575 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:08.034546    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:18:15.307211 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:15.307164    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:15.307691 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:15.307414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:15.369573 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:15.369536    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:18:15.369805 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:15.369772    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" containerID="cri-o://f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55" gracePeriod=30
Sep 16 19:18:15.369911 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:15.369817    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kube-rbac-proxy" containerID="cri-o://b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4" gracePeriod=30
Sep 16 19:18:16.383638 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:16.383606    2579 generic.go:356] "Generic (PLEG): container finished" podID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerID="b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4" exitCode=2
Sep 16 19:18:16.384010 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:16.383688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerDied","Data":"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"}
Sep 16 19:18:18.028425 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.028382    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 19:18:18.033736 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.033712    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:18.816092 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.816064    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:18:18.889651 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.889617    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config\") pod \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") "
Sep 16 19:18:18.889806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.889677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location\") pod \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") "
Sep 16 19:18:18.889806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.889708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-494jm\" (UniqueName: \"kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm\") pod \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") "
Sep 16 19:18:18.889806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.889728    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") pod \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") "
Sep 16 19:18:18.889806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.889757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert\") pod \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\" (UID: \"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0\") "
Sep 16 19:18:18.890034 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.890009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:18.890135 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.890017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:18.890218 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.890195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:18.892024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.892006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:18.892105 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.892086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm" (OuterVolumeSpecName: "kube-api-access-494jm") pod "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" (UID: "d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0"). InnerVolumeSpecName "kube-api-access-494jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:18.990693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.990619    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.990693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.990645    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.990693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.990655    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494jm\" (UniqueName: \"kubernetes.io/projected/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-kube-api-access-494jm\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.990693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.990664    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:18:18.990693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:18.990675    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:18:19.399817 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.399778    2579 generic.go:356] "Generic (PLEG): container finished" podID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerID="f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55" exitCode=0
Sep 16 19:18:19.400198 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.399828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerDied","Data":"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"}
Sep 16 19:18:19.400198 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.399841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"
Sep 16 19:18:19.400198 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.399894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b" event={"ID":"d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0","Type":"ContainerDied","Data":"3ed6090f35cceb5c072c770ca275d4ec92f90102158e95b00d60472fa891d180"}
Sep 16 19:18:19.400198 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.399931    2579 scope.go:117] "RemoveContainer" containerID="b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"
Sep 16 19:18:19.408957 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.408937    2579 scope.go:117] "RemoveContainer" containerID="f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"
Sep 16 19:18:19.416433 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.416412    2579 scope.go:117] "RemoveContainer" containerID="35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"
Sep 16 19:18:19.423637 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.423612    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:18:19.424425 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.424397    2579 scope.go:117] "RemoveContainer" containerID="b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"
Sep 16 19:18:19.424706 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:18:19.424684    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4\": container with ID starting with b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4 not found: ID does not exist" containerID="b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"
Sep 16 19:18:19.424778 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.424721    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4"} err="failed to get container status \"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4\": rpc error: code = NotFound desc = could not find container \"b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4\": container with ID starting with b9c22c5f0ce521d79ce15b20c514b256af7d66d3216e3e26b82ab2267e6080c4 not found: ID does not exist"
Sep 16 19:18:19.424778 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.424740    2579 scope.go:117] "RemoveContainer" containerID="f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"
Sep 16 19:18:19.424990 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:18:19.424971    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55\": container with ID starting with f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55 not found: ID does not exist" containerID="f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"
Sep 16 19:18:19.425047 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.424996    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55"} err="failed to get container status \"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55\": rpc error: code = NotFound desc = could not find container \"f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55\": container with ID starting with f1780735d8b396043077ed5ef6c5af89e008a9fe8ae1f238a9f5e8395a3f6c55 not found: ID does not exist"
Sep 16 19:18:19.425047 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.425010    2579 scope.go:117] "RemoveContainer" containerID="35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"
Sep 16 19:18:19.425225 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:18:19.425212    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92\": container with ID starting with 35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92 not found: ID does not exist" containerID="35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"
Sep 16 19:18:19.425263 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.425229    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92"} err="failed to get container status \"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92\": rpc error: code = NotFound desc = could not find container \"35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92\": container with ID starting with 35579ef1664ea2553a67ae655bd50fe754ecfd1e0e578d11ed819282547cae92 not found: ID does not exist"
Sep 16 19:18:19.428705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:19.428684    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-km76b"]
Sep 16 19:18:20.750007 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:20.749972    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" path="/var/lib/kubelet/pods/d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0/volumes"
Sep 16 19:18:25.307500 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:25.307422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:25.307500 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:25.307422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:35.307162 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:35.307119    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:35.307585 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:35.307439    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:45.307406 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:45.307365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:45.307916 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:45.307547    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:55.307060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:55.307020    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:18:55.307434 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:18:55.307398    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:19:05.307861 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:05.307814    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:19:05.308255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:05.307814    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:19:15.308376 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:15.308303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:19:15.308755 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:15.308393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:19:15.491450 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:15.491415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:19:15.667699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:15.667601    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container" containerID="cri-o://6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b" gracePeriod=30
Sep 16 19:19:15.667699 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:15.667651    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kube-rbac-proxy" containerID="cri-o://a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca" gracePeriod=30
Sep 16 19:19:16.675447 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:16.675416    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerID="a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca" exitCode=2
Sep 16 19:19:16.675906 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:16.675495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerDied","Data":"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"}
Sep 16 19:19:20.007926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.007902    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:19:20.110773 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.110681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location\") pod \"c2565e05-8d64-4e00-b0c7-473368ca93c4\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") "
Sep 16 19:19:20.110773 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.110738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dncz6\" (UniqueName: \"kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6\") pod \"c2565e05-8d64-4e00-b0c7-473368ca93c4\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") "
Sep 16 19:19:20.110773 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.110764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") pod \"c2565e05-8d64-4e00-b0c7-473368ca93c4\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") "
Sep 16 19:19:20.111037 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.110796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config\") pod \"c2565e05-8d64-4e00-b0c7-473368ca93c4\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") "
Sep 16 19:19:20.111037 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.110868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert\") pod \"c2565e05-8d64-4e00-b0c7-473368ca93c4\" (UID: \"c2565e05-8d64-4e00-b0c7-473368ca93c4\") "
Sep 16 19:19:20.111120 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.111097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2565e05-8d64-4e00-b0c7-473368ca93c4" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:20.111313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.111291    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2565e05-8d64-4e00-b0c7-473368ca93c4" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:20.111313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.111290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2565e05-8d64-4e00-b0c7-473368ca93c4" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:20.112966 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.112940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6" (OuterVolumeSpecName: "kube-api-access-dncz6") pod "c2565e05-8d64-4e00-b0c7-473368ca93c4" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4"). InnerVolumeSpecName "kube-api-access-dncz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:20.113068 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.113044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2565e05-8d64-4e00-b0c7-473368ca93c4" (UID: "c2565e05-8d64-4e00-b0c7-473368ca93c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:20.211991 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.211956    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:20.211991 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.211984    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2565e05-8d64-4e00-b0c7-473368ca93c4-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:20.211991 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.211997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dncz6\" (UniqueName: \"kubernetes.io/projected/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-api-access-dncz6\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:20.212227 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.212009    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2565e05-8d64-4e00-b0c7-473368ca93c4-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:20.212227 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.212021    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2565e05-8d64-4e00-b0c7-473368ca93c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:20.695228 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.695191    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerID="6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b" exitCode=0
Sep 16 19:19:20.695400 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.695248    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"
Sep 16 19:19:20.695400 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.695281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerDied","Data":"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"}
Sep 16 19:19:20.695400 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.695343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t" event={"ID":"c2565e05-8d64-4e00-b0c7-473368ca93c4","Type":"ContainerDied","Data":"1c97d29f3c601501dc3ec47f9e32413b498e1522f0d5f2d96c2bada523b4d5ec"}
Sep 16 19:19:20.695400 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.695369    2579 scope.go:117] "RemoveContainer" containerID="a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"
Sep 16 19:19:20.704944 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.704928    2579 scope.go:117] "RemoveContainer" containerID="6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"
Sep 16 19:19:20.712562 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.712542    2579 scope.go:117] "RemoveContainer" containerID="1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"
Sep 16 19:19:20.719893 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.719863    2579 scope.go:117] "RemoveContainer" containerID="a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"
Sep 16 19:19:20.720140 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:20.720118    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca\": container with ID starting with a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca not found: ID does not exist" containerID="a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"
Sep 16 19:19:20.720191 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.720147    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca"} err="failed to get container status \"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca\": rpc error: code = NotFound desc = could not find container \"a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca\": container with ID starting with a87c5fe6c56c34e7340c68ad4236b60f9ddcd61828581c1c14fe332dff9faeca not found: ID does not exist"
Sep 16 19:19:20.720191 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.720163    2579 scope.go:117] "RemoveContainer" containerID="6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"
Sep 16 19:19:20.720390 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:20.720371    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b\": container with ID starting with 6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b not found: ID does not exist" containerID="6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"
Sep 16 19:19:20.720437 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.720397    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b"} err="failed to get container status \"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b\": rpc error: code = NotFound desc = could not find container \"6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b\": container with ID starting with 6c71ce865d61f35023b3ef0c00ff1bb2c88969558ed74474ab357216f7ce826b not found: ID does not exist"
Sep 16 19:19:20.720437 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.720411    2579 scope.go:117] "RemoveContainer" containerID="1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"
Sep 16 19:19:20.720657 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:20.720639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66\": container with ID starting with 1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66 not found: ID does not exist" containerID="1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"
Sep 16 19:19:20.720712 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.720666    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66"} err="failed to get container status \"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66\": rpc error: code = NotFound desc = could not find container \"1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66\": container with ID starting with 1bf9ddb95aa47cb2defb3176ef849538d18c8567c74dbdbaa4c8456078d00d66 not found: ID does not exist"
Sep 16 19:19:20.726598 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.726576    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:19:20.732378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.732356    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f76fb68d-4dh7t"]
Sep 16 19:19:20.749243 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:20.749224    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" path="/var/lib/kubelet/pods/c2565e05-8d64-4e00-b0c7-473368ca93c4/volumes"
Sep 16 19:19:21.727474 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:21.727354    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:19:21.728007 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:21.727683    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" containerID="cri-o://78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f" gracePeriod=30
Sep 16 19:19:21.728007 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:21.727731    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kube-rbac-proxy" containerID="cri-o://5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d" gracePeriod=30
Sep 16 19:19:22.715710 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:22.715677    2579 generic.go:356] "Generic (PLEG): container finished" podID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerID="5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d" exitCode=2
Sep 16 19:19:22.715875 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:22.715739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerDied","Data":"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"}
Sep 16 19:19:25.301723 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:25.301675    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:25.307551 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:25.307517    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:19:26.275753 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.275722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:19:26.361653 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.361626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config\") pod \"79513fcc-6d6d-4280-8385-c5aae49d8f72\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") "
Sep 16 19:19:26.362000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.361676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location\") pod \"79513fcc-6d6d-4280-8385-c5aae49d8f72\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") "
Sep 16 19:19:26.362000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.361729    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w6g8\" (UniqueName: \"kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8\") pod \"79513fcc-6d6d-4280-8385-c5aae49d8f72\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") "
Sep 16 19:19:26.362000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.361752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls\") pod \"79513fcc-6d6d-4280-8385-c5aae49d8f72\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") "
Sep 16 19:19:26.362000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.361771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert\") pod \"79513fcc-6d6d-4280-8385-c5aae49d8f72\" (UID: \"79513fcc-6d6d-4280-8385-c5aae49d8f72\") "
Sep 16 19:19:26.362180 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.362020    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79513fcc-6d6d-4280-8385-c5aae49d8f72" (UID: "79513fcc-6d6d-4280-8385-c5aae49d8f72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:26.362180 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.362102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79513fcc-6d6d-4280-8385-c5aae49d8f72" (UID: "79513fcc-6d6d-4280-8385-c5aae49d8f72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:26.362253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.362175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79513fcc-6d6d-4280-8385-c5aae49d8f72" (UID: "79513fcc-6d6d-4280-8385-c5aae49d8f72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:26.364066 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.364045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79513fcc-6d6d-4280-8385-c5aae49d8f72" (UID: "79513fcc-6d6d-4280-8385-c5aae49d8f72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:26.364138 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.364116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8" (OuterVolumeSpecName: "kube-api-access-2w6g8") pod "79513fcc-6d6d-4280-8385-c5aae49d8f72" (UID: "79513fcc-6d6d-4280-8385-c5aae49d8f72"). InnerVolumeSpecName "kube-api-access-2w6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:26.462253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.462220    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6g8\" (UniqueName: \"kubernetes.io/projected/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-api-access-2w6g8\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.462253 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.462247    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79513fcc-6d6d-4280-8385-c5aae49d8f72-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.462496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.462261    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.462496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.462272    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79513fcc-6d6d-4280-8385-c5aae49d8f72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.462496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.462283    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79513fcc-6d6d-4280-8385-c5aae49d8f72-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:19:26.736057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.735972    2579 generic.go:356] "Generic (PLEG): container finished" podID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerID="78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f" exitCode=0
Sep 16 19:19:26.736057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.736027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerDied","Data":"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"}
Sep 16 19:19:26.736057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.736037    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"
Sep 16 19:19:26.736314 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.736090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd" event={"ID":"79513fcc-6d6d-4280-8385-c5aae49d8f72","Type":"ContainerDied","Data":"d7ffef9c4eec2cd7a8b2319ecc00b1ef3a6401792abfb1c1ea14b9cb7d0a3bd8"}
Sep 16 19:19:26.736314 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.736124    2579 scope.go:117] "RemoveContainer" containerID="5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"
Sep 16 19:19:26.745651 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.745634    2579 scope.go:117] "RemoveContainer" containerID="78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"
Sep 16 19:19:26.755329 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.755312    2579 scope.go:117] "RemoveContainer" containerID="f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"
Sep 16 19:19:26.762932 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.762907    2579 scope.go:117] "RemoveContainer" containerID="5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:26.763335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d\": container with ID starting with 5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d not found: ID does not exist" containerID="5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.763379    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d"} err="failed to get container status \"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d\": rpc error: code = NotFound desc = could not find container \"5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d\": container with ID starting with 5232688fc441d9de932b5560d1b1a594e42b77e383121c99581cb65c77edf96d not found: ID does not exist"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.763405    2579 scope.go:117] "RemoveContainer" containerID="78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:26.763789    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f\": container with ID starting with 78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f not found: ID does not exist" containerID="78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.763817    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f"} err="failed to get container status \"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f\": rpc error: code = NotFound desc = could not find container \"78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f\": container with ID starting with 78933b56b10074c8ff815c9197ae968c3d7db801d660373dcba66f1a7335a90f not found: ID does not exist"
Sep 16 19:19:26.763864 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.763836    2579 scope.go:117] "RemoveContainer" containerID="f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"
Sep 16 19:19:26.764238 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:19:26.764195    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593\": container with ID starting with f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593 not found: ID does not exist" containerID="f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"
Sep 16 19:19:26.764289 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.764231    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593"} err="failed to get container status \"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593\": rpc error: code = NotFound desc = could not find container \"f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593\": container with ID starting with f9cdf2a809cdb336a166c347f19fb34fb103355f50863ba3443d72fe258d6593 not found: ID does not exist"
Sep 16 19:19:26.765685 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.765666    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:19:26.774368 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:26.774338    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b684d6bb9-blgcd"]
Sep 16 19:19:28.748579 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:28.748546    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" path="/var/lib/kubelet/pods/79513fcc-6d6d-4280-8385-c5aae49d8f72/volumes"
Sep 16 19:19:32.032671 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.032627    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.032977    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.032988    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033005    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033016    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="storage-initializer"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033022    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="storage-initializer"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033028    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="storage-initializer"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033032    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="storage-initializer"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033042    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033053    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="storage-initializer"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033069    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="storage-initializer"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033078    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033093    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033098    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033153    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033161    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033167    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033173    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79513fcc-6d6d-4280-8385-c5aae49d8f72" containerName="kube-rbac-proxy"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2565e05-8d64-4e00-b0c7-473368ca93c4" containerName="kserve-container"
Sep 16 19:19:32.033347 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.033185    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86fa6c7-5256-46d9-bbed-f7eb9b70a6a0" containerName="kube-rbac-proxy"
Sep 16 19:19:32.037927 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.037907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.041034 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.040996    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 19:19:32.041178 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.041050    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 16 19:19:32.041178 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.041049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:19:32.041423 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.041407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:19:32.042243 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.042226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:19:32.042338 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.042258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:19:32.042338 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.042293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:19:32.053329 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.053307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:19:32.105760 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.105732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.105912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.105779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.105912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.105801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.105912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.105865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.105912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.105897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhhq\" (UniqueName: \"kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.206624 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.206593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.206789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.206638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.206789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.206665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.206789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.206716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.206789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.206753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhhq\" (UniqueName: \"kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.207162 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.207141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.207307 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.207281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.207350 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.207292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.209486 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.209446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.218230 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.218212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhhq\" (UniqueName: \"kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq\") pod \"isvc-lightgbm-predictor-54f85ddf88-574r7\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.348357 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.348329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:32.622059 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.622024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:19:32.624301 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:19:32.624274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2006a2_a563_4470_812c_e0eeeafd069c.slice/crio-14b30fe6e952c9aec569b2ca3821f2bc407500a98b57c7d31e4872ff4856b84d WatchSource:0}: Error finding container 14b30fe6e952c9aec569b2ca3821f2bc407500a98b57c7d31e4872ff4856b84d: Status 404 returned error can't find the container with id 14b30fe6e952c9aec569b2ca3821f2bc407500a98b57c7d31e4872ff4856b84d
Sep 16 19:19:32.761770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.761732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerStarted","Data":"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"}
Sep 16 19:19:32.761910 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:32.761801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerStarted","Data":"14b30fe6e952c9aec569b2ca3821f2bc407500a98b57c7d31e4872ff4856b84d"}
Sep 16 19:19:36.778856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:36.778766    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerID="b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175" exitCode=0
Sep 16 19:19:36.778856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:36.778815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerDied","Data":"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"}
Sep 16 19:19:36.779966 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:36.779949    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:19:43.814412 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:43.814376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerStarted","Data":"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"}
Sep 16 19:19:43.814894 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:43.814451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerStarted","Data":"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"}
Sep 16 19:19:43.814894 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:43.814675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:43.837680 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:43.837637    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podStartSLOduration=1.796425604 podStartE2EDuration="11.837622709s" totalImagesPullingTime="6.285941485s" totalInitContainerRuntime="3.75525562s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:43.835246046 +0000 UTC m=+2367.554343148" watchObservedRunningTime="2026-09-16 19:19:43.837622709 +0000 UTC m=+2367.556719785"
Sep 16 19:19:44.819651 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:44.819611    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:44.821090 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:44.821060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:19:45.830510 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:45.830444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:19:50.834774 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:50.834745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:19:50.835322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:19:50.835295    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:00.836056 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:00.836019    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:10.835693 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:10.835655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:11.086323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.086242    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:20:11.089590 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.089572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.092996 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.092968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:20:11.093232 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.093013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 19:20:11.102309 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.102287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:20:11.127286 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.127259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.127402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.127291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llsml\" (UniqueName: \"kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.127402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.127335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.127402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.127379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.127568 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.127441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.228364 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.228327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.228579 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.228379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.228579 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.228511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llsml\" (UniqueName: \"kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.228713 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.228588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.228713 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.228648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.229076 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.229043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.229172 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.229151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.229273 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.229257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.231005 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.230976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.237935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.237914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llsml\" (UniqueName: \"kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-w4qb6\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.399943 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.399855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:11.689614 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.689587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:20:11.691390 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:20:11.691361    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9f6cec_9c73_40b1_80da_d2df55c22ae2.slice/crio-cae0fee13298b1f8f4cae986ab26164e52b98aaf85bea7c7b638d3a1d44fc52b WatchSource:0}: Error finding container cae0fee13298b1f8f4cae986ab26164e52b98aaf85bea7c7b638d3a1d44fc52b: Status 404 returned error can't find the container with id cae0fee13298b1f8f4cae986ab26164e52b98aaf85bea7c7b638d3a1d44fc52b
Sep 16 19:20:11.940541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.940431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerStarted","Data":"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"}
Sep 16 19:20:11.940541 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:11.940520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerStarted","Data":"cae0fee13298b1f8f4cae986ab26164e52b98aaf85bea7c7b638d3a1d44fc52b"}
Sep 16 19:20:15.961156 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:15.961120    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerID="151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f" exitCode=0
Sep 16 19:20:15.961572 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:15.961167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerDied","Data":"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"}
Sep 16 19:20:16.968278 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:16.968244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerStarted","Data":"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"}
Sep 16 19:20:16.968707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:16.968304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerStarted","Data":"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"}
Sep 16 19:20:16.968707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:16.968576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:16.968827 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:16.968720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:16.970060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:16.970032    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:20:17.972894 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:17.972853    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:20:20.836217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:20.836183    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:22.977897 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:22.977869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:20:22.978365 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:22.978259    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:20:23.001819 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:23.001772    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podStartSLOduration=8.139193488 podStartE2EDuration="12.001757395s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862563907s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:16.995152236 +0000 UTC m=+2400.714249312" watchObservedRunningTime="2026-09-16 19:20:23.001757395 +0000 UTC m=+2406.720854473"
Sep 16 19:20:30.835980 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:30.835940    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:32.978303 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:32.978262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:20:40.835636 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:40.835554    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:42.978313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:42.978271    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:20:50.835623 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:50.835577    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:52.978578 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:20:52.978541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:00.836611 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:00.836580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:21:02.233604 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:02.233566    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:21:02.234075 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:02.233877    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container" containerID="cri-o://06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17" gracePeriod=30
Sep 16 19:21:02.234075 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:02.233909    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kube-rbac-proxy" containerID="cri-o://25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6" gracePeriod=30
Sep 16 19:21:02.979773 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:02.979736    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:03.188703 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:03.188668    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerID="25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6" exitCode=2
Sep 16 19:21:03.188871 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:03.188729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerDied","Data":"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"}
Sep 16 19:21:05.830956 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:05.830900    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:21:07.391212 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.391184    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:21:07.517936 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.517858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location\") pod \"0f2006a2-a563-4470-812c-e0eeeafd069c\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") "
Sep 16 19:21:07.517936 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.517923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls\") pod \"0f2006a2-a563-4470-812c-e0eeeafd069c\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") "
Sep 16 19:21:07.518180 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.517948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqhhq\" (UniqueName: \"kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq\") pod \"0f2006a2-a563-4470-812c-e0eeeafd069c\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") "
Sep 16 19:21:07.518180 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config\") pod \"0f2006a2-a563-4470-812c-e0eeeafd069c\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") "
Sep 16 19:21:07.518180 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert\") pod \"0f2006a2-a563-4470-812c-e0eeeafd069c\" (UID: \"0f2006a2-a563-4470-812c-e0eeeafd069c\") "
Sep 16 19:21:07.518348 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f2006a2-a563-4470-812c-e0eeeafd069c" (UID: "0f2006a2-a563-4470-812c-e0eeeafd069c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.518444 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f2006a2-a563-4470-812c-e0eeeafd069c" (UID: "0f2006a2-a563-4470-812c-e0eeeafd069c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:07.518524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f2006a2-a563-4470-812c-e0eeeafd069c" (UID: "0f2006a2-a563-4470-812c-e0eeeafd069c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:07.518524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.518508    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2006a2-a563-4470-812c-e0eeeafd069c-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.520278 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.520250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f2006a2-a563-4470-812c-e0eeeafd069c" (UID: "0f2006a2-a563-4470-812c-e0eeeafd069c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:07.520395 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.520304    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq" (OuterVolumeSpecName: "kube-api-access-lqhhq") pod "0f2006a2-a563-4470-812c-e0eeeafd069c" (UID: "0f2006a2-a563-4470-812c-e0eeeafd069c"). InnerVolumeSpecName "kube-api-access-lqhhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:07.619697 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.619663    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.619697 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.619688    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2006a2-a563-4470-812c-e0eeeafd069c-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.619697 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.619696    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhhq\" (UniqueName: \"kubernetes.io/projected/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-api-access-lqhhq\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.619918 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.619707    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2006a2-a563-4470-812c-e0eeeafd069c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.649674 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.649648    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:21:07.652028 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:07.652007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:21:08.213196 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.213164    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerID="06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17" exitCode=0
Sep 16 19:21:08.213362 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.213215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerDied","Data":"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"}
Sep 16 19:21:08.213362 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.213263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7" event={"ID":"0f2006a2-a563-4470-812c-e0eeeafd069c","Type":"ContainerDied","Data":"14b30fe6e952c9aec569b2ca3821f2bc407500a98b57c7d31e4872ff4856b84d"}
Sep 16 19:21:08.213362 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.213288    2579 scope.go:117] "RemoveContainer" containerID="25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"
Sep 16 19:21:08.213362 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.213289    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"
Sep 16 19:21:08.222856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.222838    2579 scope.go:117] "RemoveContainer" containerID="06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"
Sep 16 19:21:08.230483 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.230445    2579 scope.go:117] "RemoveContainer" containerID="b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"
Sep 16 19:21:08.237988 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.237970    2579 scope.go:117] "RemoveContainer" containerID="25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"
Sep 16 19:21:08.238230 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:08.238214    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6\": container with ID starting with 25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6 not found: ID does not exist" containerID="25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"
Sep 16 19:21:08.238280 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.238238    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6"} err="failed to get container status \"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6\": rpc error: code = NotFound desc = could not find container \"25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6\": container with ID starting with 25bd92a48317080bb85d7b8b1f8423e61e73a6cf77bf1dba40bab7018187f2b6 not found: ID does not exist"
Sep 16 19:21:08.238280 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.238257    2579 scope.go:117] "RemoveContainer" containerID="06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"
Sep 16 19:21:08.238554 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:08.238536    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17\": container with ID starting with 06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17 not found: ID does not exist" containerID="06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"
Sep 16 19:21:08.238622 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.238564    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17"} err="failed to get container status \"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17\": rpc error: code = NotFound desc = could not find container \"06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17\": container with ID starting with 06e22550b84aa2d375841b903ab646d194aed827e608295694c406ac917acd17 not found: ID does not exist"
Sep 16 19:21:08.238622 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.238580    2579 scope.go:117] "RemoveContainer" containerID="b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"
Sep 16 19:21:08.238863 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:08.238846    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175\": container with ID starting with b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175 not found: ID does not exist" containerID="b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"
Sep 16 19:21:08.238904 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.238867    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175"} err="failed to get container status \"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175\": rpc error: code = NotFound desc = could not find container \"b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175\": container with ID starting with b1c5096ede5b2b1c50130e4bb839cc00f9700523836044c114c22ed69af15175 not found: ID does not exist"
Sep 16 19:21:08.247759 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.247738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:21:08.263283 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.263264    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-574r7"]
Sep 16 19:21:08.753283 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:08.753250    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" path="/var/lib/kubelet/pods/0f2006a2-a563-4470-812c-e0eeeafd069c/volumes"
Sep 16 19:21:12.978363 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:12.978324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:22.979599 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:22.979572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:21:31.188909 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:31.188871    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:21:31.189327 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:31.189124    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" containerID="cri-o://fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de" gracePeriod=30
Sep 16 19:21:31.189327 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:31.189179    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kube-rbac-proxy" containerID="cri-o://f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491" gracePeriod=30
Sep 16 19:21:31.321413 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:31.321383    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerID="f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491" exitCode=2
Sep 16 19:21:31.321560 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:31.321476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerDied","Data":"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"}
Sep 16 19:21:32.973770 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:32.973730    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 16 19:21:32.979087 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:32.979057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:34.535289 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.535266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:21:34.640032 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.639951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location\") pod \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") "
Sep 16 19:21:34.640032 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.639990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert\") pod \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") "
Sep 16 19:21:34.640032 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls\") pod \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") "
Sep 16 19:21:34.640256 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640071    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config\") pod \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") "
Sep 16 19:21:34.640256 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llsml\" (UniqueName: \"kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml\") pod \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\" (UID: \"3e9f6cec-9c73-40b1-80da-d2df55c22ae2\") "
Sep 16 19:21:34.640369 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e9f6cec-9c73-40b1-80da-d2df55c22ae2" (UID: "3e9f6cec-9c73-40b1-80da-d2df55c22ae2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:34.640369 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640359    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:34.640513 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e9f6cec-9c73-40b1-80da-d2df55c22ae2" (UID: "3e9f6cec-9c73-40b1-80da-d2df55c22ae2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:34.640584 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.640510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e9f6cec-9c73-40b1-80da-d2df55c22ae2" (UID: "3e9f6cec-9c73-40b1-80da-d2df55c22ae2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:34.642318 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.642287    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e9f6cec-9c73-40b1-80da-d2df55c22ae2" (UID: "3e9f6cec-9c73-40b1-80da-d2df55c22ae2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:34.642449 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.642333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml" (OuterVolumeSpecName: "kube-api-access-llsml") pod "3e9f6cec-9c73-40b1-80da-d2df55c22ae2" (UID: "3e9f6cec-9c73-40b1-80da-d2df55c22ae2"). InnerVolumeSpecName "kube-api-access-llsml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:34.741246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.741212    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llsml\" (UniqueName: \"kubernetes.io/projected/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-api-access-llsml\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:34.741246 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.741242    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:34.741513 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.741253    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:34.741513 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:34.741262    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9f6cec-9c73-40b1-80da-d2df55c22ae2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:21:35.341189 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.341147    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerID="fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de" exitCode=0
Sep 16 19:21:35.341390 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.341213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"
Sep 16 19:21:35.341390 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.341222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerDied","Data":"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"}
Sep 16 19:21:35.341390 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.341299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6" event={"ID":"3e9f6cec-9c73-40b1-80da-d2df55c22ae2","Type":"ContainerDied","Data":"cae0fee13298b1f8f4cae986ab26164e52b98aaf85bea7c7b638d3a1d44fc52b"}
Sep 16 19:21:35.341390 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.341327    2579 scope.go:117] "RemoveContainer" containerID="f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"
Sep 16 19:21:35.349826 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.349811    2579 scope.go:117] "RemoveContainer" containerID="fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"
Sep 16 19:21:35.357167 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.357150    2579 scope.go:117] "RemoveContainer" containerID="151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"
Sep 16 19:21:35.364778 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.364759    2579 scope.go:117] "RemoveContainer" containerID="f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"
Sep 16 19:21:35.365195 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:35.365173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491\": container with ID starting with f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491 not found: ID does not exist" containerID="f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"
Sep 16 19:21:35.365270 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.365208    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491"} err="failed to get container status \"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491\": rpc error: code = NotFound desc = could not find container \"f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491\": container with ID starting with f28d6e4edf2514ce97c0c1e73c885514de401efcf5237bcc657585812cc96491 not found: ID does not exist"
Sep 16 19:21:35.365270 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.365232    2579 scope.go:117] "RemoveContainer" containerID="fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"
Sep 16 19:21:35.365518 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:35.365496    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de\": container with ID starting with fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de not found: ID does not exist" containerID="fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"
Sep 16 19:21:35.365583 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.365528    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de"} err="failed to get container status \"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de\": rpc error: code = NotFound desc = could not find container \"fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de\": container with ID starting with fba735849b36dec279a8448d0c6037c27eb7ec50fc06f2deb8620089571e62de not found: ID does not exist"
Sep 16 19:21:35.365583 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.365549    2579 scope.go:117] "RemoveContainer" containerID="151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"
Sep 16 19:21:35.365808 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:21:35.365783    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f\": container with ID starting with 151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f not found: ID does not exist" containerID="151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"
Sep 16 19:21:35.365853 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.365816    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f"} err="failed to get container status \"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f\": rpc error: code = NotFound desc = could not find container \"151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f\": container with ID starting with 151bf6f54c2177d910c784a636a86487dcea008ce8b40b22fa34823d5947ba5f not found: ID does not exist"
Sep 16 19:21:35.366419 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.366402    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:21:35.372820 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:35.372795    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-w4qb6"]
Sep 16 19:21:36.749243 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:21:36.749212    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" path="/var/lib/kubelet/pods/3e9f6cec-9c73-40b1-80da-d2df55c22ae2/volumes"
Sep 16 19:25:19.593887 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.593808    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594132    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="storage-initializer"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594143    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="storage-initializer"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594152    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kube-rbac-proxy"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594158    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kube-rbac-proxy"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594187    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594192    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kube-rbac-proxy"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594198    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kube-rbac-proxy"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594209    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="storage-initializer"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594214    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="storage-initializer"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594221    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594226    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594274    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594283    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kserve-container"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594290    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2006a2-a563-4470-812c-e0eeeafd069c" containerName="kube-rbac-proxy"
Sep 16 19:25:19.594322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.594296    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9f6cec-9c73-40b1-80da-d2df55c22ae2" containerName="kube-rbac-proxy"
Sep 16 19:25:19.597506 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.597490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.600080 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 19:25:19.600233 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:19.600233 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600091    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:25:19.600233 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:25:19.600233 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600110    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:19.600533 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600383    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 19:25:19.600533 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.600407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:19.609209 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.609186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:25:19.659648 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.659620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.659797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.659672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.659797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.659689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.659797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.659766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps5q\" (UniqueName: \"kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.659797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.659790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.760815 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.760777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.760815 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.760817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761085 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.760875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps5q\" (UniqueName: \"kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761085 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.760897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761085 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.760944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.761318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761642 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.761620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.761702 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.761620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.763533 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.763511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.771571 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.771548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps5q\" (UniqueName: \"kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q\") pod \"isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:19.909289 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:19.909218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:20.185792 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:20.185754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:25:20.188722 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:25:20.188698    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc9e46d_2332_4b87_8604_2096b2c8e18b.slice/crio-6d3dce866e0f8f32c7e7eb89123511bff2eb0d5ecc94621d6aa4ca607e4cb490 WatchSource:0}: Error finding container 6d3dce866e0f8f32c7e7eb89123511bff2eb0d5ecc94621d6aa4ca607e4cb490: Status 404 returned error can't find the container with id 6d3dce866e0f8f32c7e7eb89123511bff2eb0d5ecc94621d6aa4ca607e4cb490
Sep 16 19:25:20.264920 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:20.264888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerStarted","Data":"c8c4ebb481fcd1de6b25b9c587400c9d7b16e8752a2e88357722abb14c4831b2"}
Sep 16 19:25:20.265025 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:20.264962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerStarted","Data":"6d3dce866e0f8f32c7e7eb89123511bff2eb0d5ecc94621d6aa4ca607e4cb490"}
Sep 16 19:25:24.282761 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:24.282730    2579 generic.go:356] "Generic (PLEG): container finished" podID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerID="c8c4ebb481fcd1de6b25b9c587400c9d7b16e8752a2e88357722abb14c4831b2" exitCode=0
Sep 16 19:25:24.283095 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:24.282802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerDied","Data":"c8c4ebb481fcd1de6b25b9c587400c9d7b16e8752a2e88357722abb14c4831b2"}
Sep 16 19:25:24.345630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:24.345609    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:25:25.290404 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:25.290316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerStarted","Data":"d8dba42fc740190c1b227dec19fb7cac03d7f94e50a956b3b84dc86a59e28c8f"}
Sep 16 19:25:27.303534 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:27.303497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerStarted","Data":"f62ea0eeda034ff140e6c9633778328c40a2d7e95b87659087f6cae9e64a1139"}
Sep 16 19:25:27.303920 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:27.303602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:27.303920 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:27.303629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerStarted","Data":"aaf1316771fc9ecffca0a1c9c3578a68d80be58dec0778353ca95f597a963fcc"}
Sep 16 19:25:27.303920 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:27.303661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:27.331009 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:27.330963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podStartSLOduration=1.6495924020000001 podStartE2EDuration="8.330948265s" totalImagesPullingTime="2.709574746s" totalInitContainerRuntime="3.971781117s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:27.328401263 +0000 UTC m=+2711.047498342" watchObservedRunningTime="2026-09-16 19:25:27.330948265 +0000 UTC m=+2711.050045341"
Sep 16 19:25:28.308399 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:28.308362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:34.318121 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:34.318091    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:25:54.319051 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:25:54.319013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:26:04.319927 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:04.319897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:26:07.680639 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:07.680613    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:26:07.684126 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:07.684105    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:26:34.320640 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:34.320605    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:26:39.666707 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:39.666629    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:26:39.669083 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:39.666885    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container" containerID="cri-o://d8dba42fc740190c1b227dec19fb7cac03d7f94e50a956b3b84dc86a59e28c8f" gracePeriod=30
Sep 16 19:26:39.669083 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:39.666926    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-agent" containerID="cri-o://aaf1316771fc9ecffca0a1c9c3578a68d80be58dec0778353ca95f597a963fcc" gracePeriod=30
Sep 16 19:26:39.669083 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:39.666941    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" containerID="cri-o://f62ea0eeda034ff140e6c9633778328c40a2d7e95b87659087f6cae9e64a1139" gracePeriod=30
Sep 16 19:26:40.623588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:40.623553    2579 generic.go:356] "Generic (PLEG): container finished" podID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerID="f62ea0eeda034ff140e6c9633778328c40a2d7e95b87659087f6cae9e64a1139" exitCode=2
Sep 16 19:26:40.623588 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:40.623588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerDied","Data":"f62ea0eeda034ff140e6c9633778328c40a2d7e95b87659087f6cae9e64a1139"}
Sep 16 19:26:42.635050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:42.635013    2579 generic.go:356] "Generic (PLEG): container finished" podID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerID="d8dba42fc740190c1b227dec19fb7cac03d7f94e50a956b3b84dc86a59e28c8f" exitCode=0
Sep 16 19:26:42.635050 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:42.635055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerDied","Data":"d8dba42fc740190c1b227dec19fb7cac03d7f94e50a956b3b84dc86a59e28c8f"}
Sep 16 19:26:44.313043 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:44.313003    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:26:44.318874 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:44.318845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:26:49.313152 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:49.313102    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:26:54.313084 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:54.313042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:26:54.313584 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:54.313224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:26:54.319212 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:54.319179    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:26:59.313668 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:26:59.313622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:27:04.312888 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:04.312841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:27:04.319288 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:04.319254    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:27:04.319383 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:04.319373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:27:08.360185 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.360150    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:08.363805 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.363785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.366402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.366378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 16 19:27:08.366511 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.366386    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:27:08.366575 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.366487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:27:08.374992 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.374971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:08.464346 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.464315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.464560 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.464363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.464560 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.464430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpjh\" (UniqueName: \"kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.464560 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.464469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.565361 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.565326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.565361 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.565366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.565641 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.565394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpjh\" (UniqueName: \"kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.565641 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.565417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.565830 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.565809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.566034 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.566015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.568033 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.568011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.573989 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.573965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpjh\" (UniqueName: \"kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.674484 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.674397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:08.948493 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:08.948448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:08.950609 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:27:08.950579    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4989c5_191b_49ce_8a00_0896badabb0c.slice/crio-4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc WatchSource:0}: Error finding container 4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc: Status 404 returned error can't find the container with id 4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc
Sep 16 19:27:09.313715 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.313674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:27:09.753099 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.753065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerStarted","Data":"b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"}
Sep 16 19:27:09.753549 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.753151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerStarted","Data":"4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc"}
Sep 16 19:27:09.757445 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.757398    2579 generic.go:356] "Generic (PLEG): container finished" podID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerID="aaf1316771fc9ecffca0a1c9c3578a68d80be58dec0778353ca95f597a963fcc" exitCode=137
Sep 16 19:27:09.757593 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.757441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerDied","Data":"aaf1316771fc9ecffca0a1c9c3578a68d80be58dec0778353ca95f597a963fcc"}
Sep 16 19:27:09.820562 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.820538    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:27:09.981328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tps5q\" (UniqueName: \"kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q\") pod \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") "
Sep 16 19:27:09.981328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls\") pod \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") "
Sep 16 19:27:09.981328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config\") pod \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") "
Sep 16 19:27:09.981328 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert\") pod \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") "
Sep 16 19:27:09.981698 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location\") pod \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\" (UID: \"cbc9e46d-2332-4b87-8604-2096b2c8e18b\") "
Sep 16 19:27:09.981756 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981692    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbc9e46d-2332-4b87-8604-2096b2c8e18b" (UID: "cbc9e46d-2332-4b87-8604-2096b2c8e18b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:09.981756 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbc9e46d-2332-4b87-8604-2096b2c8e18b" (UID: "cbc9e46d-2332-4b87-8604-2096b2c8e18b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:09.981827 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.981765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbc9e46d-2332-4b87-8604-2096b2c8e18b" (UID: "cbc9e46d-2332-4b87-8604-2096b2c8e18b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:09.983540 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.983517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbc9e46d-2332-4b87-8604-2096b2c8e18b" (UID: "cbc9e46d-2332-4b87-8604-2096b2c8e18b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:09.983690 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:09.983670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q" (OuterVolumeSpecName: "kube-api-access-tps5q") pod "cbc9e46d-2332-4b87-8604-2096b2c8e18b" (UID: "cbc9e46d-2332-4b87-8604-2096b2c8e18b"). InnerVolumeSpecName "kube-api-access-tps5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:10.082034 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.081995    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tps5q\" (UniqueName: \"kubernetes.io/projected/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-api-access-tps5q\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.082034 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.082032    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc9e46d-2332-4b87-8604-2096b2c8e18b-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.082250 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.082048    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.082250 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.082060    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc9e46d-2332-4b87-8604-2096b2c8e18b-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.082250 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.082072    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc9e46d-2332-4b87-8604-2096b2c8e18b-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:10.764308 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.764285    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"
Sep 16 19:27:10.764668 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.764308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n" event={"ID":"cbc9e46d-2332-4b87-8604-2096b2c8e18b","Type":"ContainerDied","Data":"6d3dce866e0f8f32c7e7eb89123511bff2eb0d5ecc94621d6aa4ca607e4cb490"}
Sep 16 19:27:10.764668 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.764385    2579 scope.go:117] "RemoveContainer" containerID="f62ea0eeda034ff140e6c9633778328c40a2d7e95b87659087f6cae9e64a1139"
Sep 16 19:27:10.773069 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.773051    2579 scope.go:117] "RemoveContainer" containerID="aaf1316771fc9ecffca0a1c9c3578a68d80be58dec0778353ca95f597a963fcc"
Sep 16 19:27:10.780589 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.780568    2579 scope.go:117] "RemoveContainer" containerID="d8dba42fc740190c1b227dec19fb7cac03d7f94e50a956b3b84dc86a59e28c8f"
Sep 16 19:27:10.786283 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.786257    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:27:10.788935 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.788914    2579 scope.go:117] "RemoveContainer" containerID="c8c4ebb481fcd1de6b25b9c587400c9d7b16e8752a2e88357722abb14c4831b2"
Sep 16 19:27:10.790372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:10.790351    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6d944dbd5f-jc45n"]
Sep 16 19:27:12.748824 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:12.748792    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" path="/var/lib/kubelet/pods/cbc9e46d-2332-4b87-8604-2096b2c8e18b/volumes"
Sep 16 19:27:13.780051 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:13.780025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/0.log"
Sep 16 19:27:13.780401 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:13.780061    2579 generic.go:356] "Generic (PLEG): container finished" podID="af4989c5-191b-49ce-8a00-0896badabb0c" containerID="b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423" exitCode=1
Sep 16 19:27:13.780401 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:13.780094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerDied","Data":"b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"}
Sep 16 19:27:14.786801 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:14.786778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/0.log"
Sep 16 19:27:14.787188 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:14.786855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerStarted","Data":"e4abf48c7ba6441a1626daafb916d213a0b5ac44d5962c6f5fc7e4f8f7870399"}
Sep 16 19:27:16.796518 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.796491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/1.log"
Sep 16 19:27:16.796941 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.796864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/0.log"
Sep 16 19:27:16.796941 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.796896    2579 generic.go:356] "Generic (PLEG): container finished" podID="af4989c5-191b-49ce-8a00-0896badabb0c" containerID="e4abf48c7ba6441a1626daafb916d213a0b5ac44d5962c6f5fc7e4f8f7870399" exitCode=1
Sep 16 19:27:16.797025 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.796967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerDied","Data":"e4abf48c7ba6441a1626daafb916d213a0b5ac44d5962c6f5fc7e4f8f7870399"}
Sep 16 19:27:16.797061 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.797032    2579 scope.go:117] "RemoveContainer" containerID="b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"
Sep 16 19:27:16.797413 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.797394    2579 scope.go:117] "RemoveContainer" containerID="b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"
Sep 16 19:27:16.807977 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:27:16.807957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_kserve-ci-e2e-test(af4989c5-191b-49ce-8a00-0896badabb0c)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" podUID="af4989c5-191b-49ce-8a00-0896badabb0c"
Sep 16 19:27:16.808112 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:27:16.808088    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_kserve-ci-e2e-test_af4989c5-191b-49ce-8a00-0896badabb0c_0 in pod sandbox 4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc from index: no such id: 'b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423'" containerID="b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"
Sep 16 19:27:16.808168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:16.808123    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_kserve-ci-e2e-test_af4989c5-191b-49ce-8a00-0896badabb0c_0 in pod sandbox 4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc from index: no such id: 'b1868d288615b69cde1b7a5cae11ecc738ab68f624e77cc3c0c2fb0b8636e423'"
Sep 16 19:27:17.803747 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:17.803717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/1.log"
Sep 16 19:27:18.411764 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.411731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:18.541739 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.541720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/1.log"
Sep 16 19:27:18.545140 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.545124    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:18.654666 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.654636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vpjh\" (UniqueName: \"kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh\") pod \"af4989c5-191b-49ce-8a00-0896badabb0c\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") "
Sep 16 19:27:18.654787 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.654721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls\") pod \"af4989c5-191b-49ce-8a00-0896badabb0c\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") "
Sep 16 19:27:18.654787 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.654747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location\") pod \"af4989c5-191b-49ce-8a00-0896badabb0c\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") "
Sep 16 19:27:18.654787 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.654766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config\") pod \"af4989c5-191b-49ce-8a00-0896badabb0c\" (UID: \"af4989c5-191b-49ce-8a00-0896badabb0c\") "
Sep 16 19:27:18.655031 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.654987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af4989c5-191b-49ce-8a00-0896badabb0c" (UID: "af4989c5-191b-49ce-8a00-0896badabb0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:18.655200 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.655170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af4989c5-191b-49ce-8a00-0896badabb0c" (UID: "af4989c5-191b-49ce-8a00-0896badabb0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:18.656804 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.656782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh" (OuterVolumeSpecName: "kube-api-access-9vpjh") pod "af4989c5-191b-49ce-8a00-0896badabb0c" (UID: "af4989c5-191b-49ce-8a00-0896badabb0c"). InnerVolumeSpecName "kube-api-access-9vpjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:18.656872 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.656812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af4989c5-191b-49ce-8a00-0896badabb0c" (UID: "af4989c5-191b-49ce-8a00-0896badabb0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:18.756231 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.756170    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpjh\" (UniqueName: \"kubernetes.io/projected/af4989c5-191b-49ce-8a00-0896badabb0c-kube-api-access-9vpjh\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.756231 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.756194    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4989c5-191b-49ce-8a00-0896badabb0c-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.756231 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.756204    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4989c5-191b-49ce-8a00-0896badabb0c-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.756231 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.756213    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4989c5-191b-49ce-8a00-0896badabb0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.810121 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.810096    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k_af4989c5-191b-49ce-8a00-0896badabb0c/storage-initializer/1.log"
Sep 16 19:27:18.810494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.810171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k" event={"ID":"af4989c5-191b-49ce-8a00-0896badabb0c","Type":"ContainerDied","Data":"4904f504592f443418e25ce93f5ec0977ef00ed5338aaffa64895046621c55cc"}
Sep 16 19:27:18.810494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.810187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"
Sep 16 19:27:18.810494 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.810249    2579 scope.go:117] "RemoveContainer" containerID="e4abf48c7ba6441a1626daafb916d213a0b5ac44d5962c6f5fc7e4f8f7870399"
Sep 16 19:27:18.846344 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.846318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:18.851255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:18.851234    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-74d78678c6-tjf7k"]
Sep 16 19:27:19.509104 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509065    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:27:19.509482 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509443    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy"
Sep 16 19:27:19.509482 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509479    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509514    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="storage-initializer"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="storage-initializer"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509534    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-agent"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-agent"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509561    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509572    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container"
Sep 16 19:27:19.509630 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509676    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509688    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-container"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509697    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kube-rbac-proxy"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509707    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc9e46d-2332-4b87-8604-2096b2c8e18b" containerName="kserve-agent"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509716    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509802    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.510101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.509813    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" containerName="storage-initializer"
Sep 16 19:27:19.514142 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.514120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.517112 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.517086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:27:19.517241 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.517121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:19.517241 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.517142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:27:19.517241 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.517166    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:27:19.518268 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.518252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 16 19:27:19.518374 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.518285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:27:19.518374 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.518298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:27:19.524634 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.524612    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:27:19.562345 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.562315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.562486 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.562365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxdl\" (UniqueName: \"kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.562668 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.562637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.562722 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.562700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.562781 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.562755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663280 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663407 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663407 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663407 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663407 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxdl\" (UniqueName: \"kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.663841 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.663818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.664122 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.664100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.664185 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.664100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.666100 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.666082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.672694 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.672669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxdl\" (UniqueName: \"kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:19.825348 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:19.825318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:20.099606 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:20.099577    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:27:20.102133 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:27:20.102098    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cf9c97_2fe1_4be1_8b7f_89b51a601824.slice/crio-93f4b6628dca43599756799511bee7fa08f6e7e458e346f92504b7805878eca5 WatchSource:0}: Error finding container 93f4b6628dca43599756799511bee7fa08f6e7e458e346f92504b7805878eca5: Status 404 returned error can't find the container with id 93f4b6628dca43599756799511bee7fa08f6e7e458e346f92504b7805878eca5
Sep 16 19:27:20.748829 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:20.748797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4989c5-191b-49ce-8a00-0896badabb0c" path="/var/lib/kubelet/pods/af4989c5-191b-49ce-8a00-0896badabb0c/volumes"
Sep 16 19:27:20.821880 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:20.821843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerStarted","Data":"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"}
Sep 16 19:27:20.822025 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:20.821911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerStarted","Data":"93f4b6628dca43599756799511bee7fa08f6e7e458e346f92504b7805878eca5"}
Sep 16 19:27:21.828258 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:21.828226    2579 generic.go:356] "Generic (PLEG): container finished" podID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerID="3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367" exitCode=0
Sep 16 19:27:21.828667 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:21.828315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerDied","Data":"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"}
Sep 16 19:27:22.835088 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:22.835046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerStarted","Data":"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"}
Sep 16 19:27:22.835550 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:22.835102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerStarted","Data":"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"}
Sep 16 19:27:22.835550 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:22.835129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:22.854890 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:22.854848    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podStartSLOduration=2.771860556 podStartE2EDuration="3.854832333s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.082971777s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:22.853003099 +0000 UTC m=+2826.572100201" watchObservedRunningTime="2026-09-16 19:27:22.854832333 +0000 UTC m=+2826.573929410"
Sep 16 19:27:23.840234 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:23.840203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:23.841443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:23.841414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:27:24.843988 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:24.843948    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:27:29.849938 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:29.849906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:27:29.850594 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:29.850563    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:27:39.850780 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:39.850739    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:27:49.851294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:49.851252    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:27:59.851221 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:27:59.851176    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:28:09.850639 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:09.850559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:28:19.851025 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:19.850987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:28:29.850595 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:29.850557    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:28:39.851177 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:39.851150    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:28:49.520320 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:49.520287    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:28:49.520744 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:49.520537    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" containerID="cri-o://b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc" gracePeriod=30
Sep 16 19:28:49.520744 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:49.520648    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kube-rbac-proxy" containerID="cri-o://6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb" gracePeriod=30
Sep 16 19:28:49.845136 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:49.845089    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 16 19:28:49.850536 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:49.850505    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:28:50.204184 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.204106    2579 generic.go:356] "Generic (PLEG): container finished" podID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerID="6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb" exitCode=2
Sep 16 19:28:50.204184 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.204154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerDied","Data":"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"}
Sep 16 19:28:50.709048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.709014    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:28:50.712537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.712519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.714965 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.714942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:28:50.715058 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.714990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 19:28:50.724228 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.724206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:28:50.784161 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.784130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.784298 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.784173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.784298 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.784197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp55\" (UniqueName: \"kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.784298 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.784265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885087 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp55\" (UniqueName: \"kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885526 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.885805 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.885780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.887875 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.887850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:50.896666 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:50.896645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp55\" (UniqueName: \"kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:51.022806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:51.022710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:28:51.294765 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:51.294742    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:28:51.296835 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:28:51.296808    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fee15ff_479c_46c2_9801_1d30c6129c7b.slice/crio-856e836166c7f7aa892981b7224212b5085ee96b8c8412c0e8876eae4c29c1d6 WatchSource:0}: Error finding container 856e836166c7f7aa892981b7224212b5085ee96b8c8412c0e8876eae4c29c1d6: Status 404 returned error can't find the container with id 856e836166c7f7aa892981b7224212b5085ee96b8c8412c0e8876eae4c29c1d6
Sep 16 19:28:52.215684 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:52.215650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerStarted","Data":"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"}
Sep 16 19:28:52.216101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:52.215701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerStarted","Data":"856e836166c7f7aa892981b7224212b5085ee96b8c8412c0e8876eae4c29c1d6"}
Sep 16 19:28:53.862352 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.862329    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:28:53.912405 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config\") pod \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") "
Sep 16 19:28:53.912405 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location\") pod \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") "
Sep 16 19:28:53.912653 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sxdl\" (UniqueName: \"kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl\") pod \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") "
Sep 16 19:28:53.912653 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls\") pod \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") "
Sep 16 19:28:53.912653 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert\") pod \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\" (UID: \"71cf9c97-2fe1-4be1-8b7f-89b51a601824\") "
Sep 16 19:28:53.912812 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71cf9c97-2fe1-4be1-8b7f-89b51a601824" (UID: "71cf9c97-2fe1-4be1-8b7f-89b51a601824"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:53.912812 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71cf9c97-2fe1-4be1-8b7f-89b51a601824" (UID: "71cf9c97-2fe1-4be1-8b7f-89b51a601824"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:53.912912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912846    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:28:53.912912 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912864    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:28:53.912981 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.912911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71cf9c97-2fe1-4be1-8b7f-89b51a601824" (UID: "71cf9c97-2fe1-4be1-8b7f-89b51a601824"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:53.914571 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.914545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl" (OuterVolumeSpecName: "kube-api-access-6sxdl") pod "71cf9c97-2fe1-4be1-8b7f-89b51a601824" (UID: "71cf9c97-2fe1-4be1-8b7f-89b51a601824"). InnerVolumeSpecName "kube-api-access-6sxdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:53.914696 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:53.914655    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71cf9c97-2fe1-4be1-8b7f-89b51a601824" (UID: "71cf9c97-2fe1-4be1-8b7f-89b51a601824"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:54.014238 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.014204    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxdl\" (UniqueName: \"kubernetes.io/projected/71cf9c97-2fe1-4be1-8b7f-89b51a601824-kube-api-access-6sxdl\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:28:54.014238 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.014230    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cf9c97-2fe1-4be1-8b7f-89b51a601824-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:28:54.014238 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.014242    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71cf9c97-2fe1-4be1-8b7f-89b51a601824-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:28:54.229858 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.229769    2579 generic.go:356] "Generic (PLEG): container finished" podID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerID="b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc" exitCode=0
Sep 16 19:28:54.229858 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.229842    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"
Sep 16 19:28:54.230089 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.229842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerDied","Data":"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"}
Sep 16 19:28:54.230089 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.230003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6" event={"ID":"71cf9c97-2fe1-4be1-8b7f-89b51a601824","Type":"ContainerDied","Data":"93f4b6628dca43599756799511bee7fa08f6e7e458e346f92504b7805878eca5"}
Sep 16 19:28:54.230089 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.230051    2579 scope.go:117] "RemoveContainer" containerID="6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"
Sep 16 19:28:54.239239 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.239225    2579 scope.go:117] "RemoveContainer" containerID="b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"
Sep 16 19:28:54.246898 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.246881    2579 scope.go:117] "RemoveContainer" containerID="3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"
Sep 16 19:28:54.254888 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.254866    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:28:54.256065 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256037    2579 scope.go:117] "RemoveContainer" containerID="6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"
Sep 16 19:28:54.256331 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:28:54.256313    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb\": container with ID starting with 6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb not found: ID does not exist" containerID="6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"
Sep 16 19:28:54.256410 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256343    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb"} err="failed to get container status \"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb\": rpc error: code = NotFound desc = could not find container \"6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb\": container with ID starting with 6cbcf9935af07c192eae2a8bbccde35e3541dc43b5c9d3b46ffb4d73dd3d81eb not found: ID does not exist"
Sep 16 19:28:54.256410 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256369    2579 scope.go:117] "RemoveContainer" containerID="b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"
Sep 16 19:28:54.256656 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:28:54.256639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc\": container with ID starting with b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc not found: ID does not exist" containerID="b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"
Sep 16 19:28:54.256705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256664    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc"} err="failed to get container status \"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc\": rpc error: code = NotFound desc = could not find container \"b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc\": container with ID starting with b9bc33dc1b14630c66900a2f55da9728e338792b82967e769c99c6b1588187fc not found: ID does not exist"
Sep 16 19:28:54.256705 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256680    2579 scope.go:117] "RemoveContainer" containerID="3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"
Sep 16 19:28:54.256943 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256924    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7d549cc95b-vwtc6"]
Sep 16 19:28:54.256990 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:28:54.256947    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367\": container with ID starting with 3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367 not found: ID does not exist" containerID="3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"
Sep 16 19:28:54.256990 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.256967    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367"} err="failed to get container status \"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367\": rpc error: code = NotFound desc = could not find container \"3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367\": container with ID starting with 3352b7a6faf59b766db4bcdb6526ac36d1510301a03b546ef9884d4311024367 not found: ID does not exist"
Sep 16 19:28:54.752883 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:54.750268    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" path="/var/lib/kubelet/pods/71cf9c97-2fe1-4be1-8b7f-89b51a601824/volumes"
Sep 16 19:28:57.245918 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:57.245894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/0.log"
Sep 16 19:28:57.246213 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:57.245931    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerID="8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995" exitCode=1
Sep 16 19:28:57.246213 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:57.245985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerDied","Data":"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"}
Sep 16 19:28:58.252377 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:58.252350    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/0.log"
Sep 16 19:28:58.252765 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:28:58.252425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerStarted","Data":"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"}
Sep 16 19:29:00.630571 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.630536    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:29:00.630986 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.630729    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer" containerID="cri-o://ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf" gracePeriod=30
Sep 16 19:29:00.760069 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.760047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/1.log"
Sep 16 19:29:00.760505 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.760488    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/0.log"
Sep 16 19:29:00.762694 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.762679    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:29:00.865681 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.865651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls\") pod \"9fee15ff-479c-46c2-9801-1d30c6129c7b\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") "
Sep 16 19:29:00.865823 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.865728    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lp55\" (UniqueName: \"kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55\") pod \"9fee15ff-479c-46c2-9801-1d30c6129c7b\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") "
Sep 16 19:29:00.865823 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.865765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location\") pod \"9fee15ff-479c-46c2-9801-1d30c6129c7b\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") "
Sep 16 19:29:00.865823 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.865800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config\") pod \"9fee15ff-479c-46c2-9801-1d30c6129c7b\" (UID: \"9fee15ff-479c-46c2-9801-1d30c6129c7b\") "
Sep 16 19:29:00.866254 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.866029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fee15ff-479c-46c2-9801-1d30c6129c7b" (UID: "9fee15ff-479c-46c2-9801-1d30c6129c7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:00.866254 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.866125    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fee15ff-479c-46c2-9801-1d30c6129c7b-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:29:00.866254 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.866177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fee15ff-479c-46c2-9801-1d30c6129c7b" (UID: "9fee15ff-479c-46c2-9801-1d30c6129c7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:00.867847 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.867829    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fee15ff-479c-46c2-9801-1d30c6129c7b" (UID: "9fee15ff-479c-46c2-9801-1d30c6129c7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:00.868008 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.867989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55" (OuterVolumeSpecName: "kube-api-access-6lp55") pod "9fee15ff-479c-46c2-9801-1d30c6129c7b" (UID: "9fee15ff-479c-46c2-9801-1d30c6129c7b"). InnerVolumeSpecName "kube-api-access-6lp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:00.966496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.966415    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fee15ff-479c-46c2-9801-1d30c6129c7b-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:29:00.966496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.966438    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lp55\" (UniqueName: \"kubernetes.io/projected/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-api-access-6lp55\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:29:00.966496 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:00.966446    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fee15ff-479c-46c2-9801-1d30c6129c7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.266840 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.266766    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/1.log"
Sep 16 19:29:01.267141 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267125    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v_9fee15ff-479c-46c2-9801-1d30c6129c7b/storage-initializer/0.log"
Sep 16 19:29:01.267198 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267165    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerID="ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf" exitCode=1
Sep 16 19:29:01.267237 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267222    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"
Sep 16 19:29:01.267269 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerDied","Data":"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"}
Sep 16 19:29:01.267313 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v" event={"ID":"9fee15ff-479c-46c2-9801-1d30c6129c7b","Type":"ContainerDied","Data":"856e836166c7f7aa892981b7224212b5085ee96b8c8412c0e8876eae4c29c1d6"}
Sep 16 19:29:01.267359 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.267335    2579 scope.go:117] "RemoveContainer" containerID="ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"
Sep 16 19:29:01.282202 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.282181    2579 scope.go:117] "RemoveContainer" containerID="8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"
Sep 16 19:29:01.289844 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.289826    2579 scope.go:117] "RemoveContainer" containerID="ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"
Sep 16 19:29:01.290118 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:29:01.290100    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf\": container with ID starting with ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf not found: ID does not exist" containerID="ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"
Sep 16 19:29:01.290195 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.290127    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf"} err="failed to get container status \"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf\": rpc error: code = NotFound desc = could not find container \"ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf\": container with ID starting with ae5d836bea70bb5d3f339948fc7494c9d8c4f9ed47264e3ffdb8f3650b6c6aaf not found: ID does not exist"
Sep 16 19:29:01.290195 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.290144    2579 scope.go:117] "RemoveContainer" containerID="8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"
Sep 16 19:29:01.290366 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:29:01.290349    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995\": container with ID starting with 8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995 not found: ID does not exist" containerID="8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"
Sep 16 19:29:01.290407 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.290372    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995"} err="failed to get container status \"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995\": rpc error: code = NotFound desc = could not find container \"8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995\": container with ID starting with 8af5731fe90588af1c02eeb1e0fdd0df1c88c50310c31779fff1450144d07995 not found: ID does not exist"
Sep 16 19:29:01.310700 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.310669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:29:01.312188 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.312165    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55bf49f9d4-vz67v"]
Sep 16 19:29:01.749244 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749604    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749617    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="storage-initializer"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="storage-initializer"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749630    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kube-rbac-proxy"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749635    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kube-rbac-proxy"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749658    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.749683 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.749992 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749711    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.749992 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kserve-container"
Sep 16 19:29:01.749992 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cf9c97-2fe1-4be1-8b7f-89b51a601824" containerName="kube-rbac-proxy"
Sep 16 19:29:01.749992 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.749827    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" containerName="storage-initializer"
Sep 16 19:29:01.754304 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.754284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.757237 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:29:01.757237 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757221    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:01.757472 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757260    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:29:01.757472 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757214    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:01.757472 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:01.757610 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.757571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 19:29:01.758829 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.758808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:29:01.764324 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.764303    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:29:01.872744 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.872710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.872926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.872795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.872926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.872822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.872926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.872850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.872926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.872872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5v5\" (UniqueName: \"kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974264 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974264 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974264 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5v5\" (UniqueName: \"kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974264 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974573 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974865 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.974959 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.974896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.976716 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.976699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:01.983677 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:01.983658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5v5\" (UniqueName: \"kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:02.065629 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:02.065590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:02.350717 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:02.350696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:29:02.352498 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:29:02.352472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c17a16e_bf0b_42c7_b0bc_e5993c6a49bf.slice/crio-5d2809f88fe57f442927fa6d3d9d1dc4fb27a9fb02a7aa12c803ab8124108754 WatchSource:0}: Error finding container 5d2809f88fe57f442927fa6d3d9d1dc4fb27a9fb02a7aa12c803ab8124108754: Status 404 returned error can't find the container with id 5d2809f88fe57f442927fa6d3d9d1dc4fb27a9fb02a7aa12c803ab8124108754
Sep 16 19:29:02.750273 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:02.750190    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fee15ff-479c-46c2-9801-1d30c6129c7b" path="/var/lib/kubelet/pods/9fee15ff-479c-46c2-9801-1d30c6129c7b/volumes"
Sep 16 19:29:03.279345 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:03.279308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerStarted","Data":"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"}
Sep 16 19:29:03.279537 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:03.279377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerStarted","Data":"5d2809f88fe57f442927fa6d3d9d1dc4fb27a9fb02a7aa12c803ab8124108754"}
Sep 16 19:29:04.285691 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:04.285658    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerID="0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee" exitCode=0
Sep 16 19:29:04.286069 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:04.285697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerDied","Data":"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"}
Sep 16 19:29:05.292173 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:05.292136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerStarted","Data":"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"}
Sep 16 19:29:05.292586 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:05.292220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerStarted","Data":"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"}
Sep 16 19:29:05.292586 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:05.292248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:05.313224 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:05.313177    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podStartSLOduration=3.241072148 podStartE2EDuration="4.31316164s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.072089492s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:05.312530009 +0000 UTC m=+2929.031627087" watchObservedRunningTime="2026-09-16 19:29:05.31316164 +0000 UTC m=+2929.032258718"
Sep 16 19:29:06.296969 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:06.296939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:06.298214 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:06.298185    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:07.301871 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:07.301830    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:12.306729 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:12.306702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:29:12.307168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:12.307144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:22.307787 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:22.307748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:32.307233 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:32.307193    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:42.307386 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:42.307298    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:29:52.308092 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:29:52.308051    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:30:02.308100 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:02.308057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:30:12.307970 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:12.307924    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:30:22.308210 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:22.308179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:30:31.802402 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:31.802367    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:30:31.802959 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:31.802710    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" containerID="cri-o://fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822" gracePeriod=30
Sep 16 19:30:31.802959 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:31.802721    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kube-rbac-proxy" containerID="cri-o://e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44" gracePeriod=30
Sep 16 19:30:32.302502 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:32.302471    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 16 19:30:32.307792 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:32.307766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:30:32.683928 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:32.683838    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerID="e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44" exitCode=2
Sep 16 19:30:32.683928 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:32.683914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerDied","Data":"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"}
Sep 16 19:30:36.244535 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.244509    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:30:36.266628 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert\") pod \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") "
Sep 16 19:30:36.266748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location\") pod \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") "
Sep 16 19:30:36.266748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj5v5\" (UniqueName: \"kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5\") pod \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") "
Sep 16 19:30:36.266748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls\") pod \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") "
Sep 16 19:30:36.266926 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config\") pod \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\" (UID: \"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf\") "
Sep 16 19:30:36.266989 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.266963    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" (UID: "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:36.267301 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.267278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" (UID: "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:36.267562 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.267523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" (UID: "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:36.269653 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.269628    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" (UID: "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:36.269754 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.269729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5" (OuterVolumeSpecName: "kube-api-access-qj5v5") pod "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" (UID: "5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf"). InnerVolumeSpecName "kube-api-access-qj5v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:36.368299 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.368277    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5v5\" (UniqueName: \"kubernetes.io/projected/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-api-access-qj5v5\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:30:36.368299 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.368298    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:30:36.368450 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.368308    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:30:36.368450 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.368316    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:30:36.368450 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.368324    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:30:36.709024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.708930    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerID="fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822" exitCode=0
Sep 16 19:30:36.709024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.709009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"
Sep 16 19:30:36.709024 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.709011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerDied","Data":"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"}
Sep 16 19:30:36.709249 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.709093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42" event={"ID":"5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf","Type":"ContainerDied","Data":"5d2809f88fe57f442927fa6d3d9d1dc4fb27a9fb02a7aa12c803ab8124108754"}
Sep 16 19:30:36.709249 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.709122    2579 scope.go:117] "RemoveContainer" containerID="e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"
Sep 16 19:30:36.718372 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.718350    2579 scope.go:117] "RemoveContainer" containerID="fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"
Sep 16 19:30:36.726717 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.726701    2579 scope.go:117] "RemoveContainer" containerID="0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"
Sep 16 19:30:36.732788 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.732763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:30:36.734643 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.734619    2579 scope.go:117] "RemoveContainer" containerID="e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"
Sep 16 19:30:36.734920 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:30:36.734900    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44\": container with ID starting with e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44 not found: ID does not exist" containerID="e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"
Sep 16 19:30:36.734973 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.734928    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44"} err="failed to get container status \"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44\": rpc error: code = NotFound desc = could not find container \"e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44\": container with ID starting with e9ebdc13d5faaab8f9357d888d7eb61348cb224a7f7b709e147f09cd702d8a44 not found: ID does not exist"
Sep 16 19:30:36.734973 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.734944    2579 scope.go:117] "RemoveContainer" containerID="fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"
Sep 16 19:30:36.735208 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:30:36.735192    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822\": container with ID starting with fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822 not found: ID does not exist" containerID="fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"
Sep 16 19:30:36.735247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.735215    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822"} err="failed to get container status \"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822\": rpc error: code = NotFound desc = could not find container \"fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822\": container with ID starting with fff04e97a247c8205438aed4b4589cd8a4ed8cc975a6ee49380b18500438a822 not found: ID does not exist"
Sep 16 19:30:36.735247 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.735229    2579 scope.go:117] "RemoveContainer" containerID="0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"
Sep 16 19:30:36.735506 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:30:36.735481    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee\": container with ID starting with 0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee not found: ID does not exist" containerID="0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"
Sep 16 19:30:36.735558 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.735514    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee"} err="failed to get container status \"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee\": rpc error: code = NotFound desc = could not find container \"0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee\": container with ID starting with 0bfd8200e9100087b6abb7328f93041ae3cb28e5b2f088a01474146e45c74aee not found: ID does not exist"
Sep 16 19:30:36.739524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.739501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74d6cf484c-jrx42"]
Sep 16 19:30:36.749009 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:30:36.748989    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" path="/var/lib/kubelet/pods/5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf/volumes"
Sep 16 19:31:07.711755 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:31:07.711678    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:31:07.720733 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:31:07.720714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:32:00.579048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579013    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579335    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579346    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579363    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="storage-initializer"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="storage-initializer"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579378    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kube-rbac-proxy"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kube-rbac-proxy"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579443    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kube-rbac-proxy"
Sep 16 19:32:00.581271 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.579464    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c17a16e-bf0b-42c7-b0bc-e5993c6a49bf" containerName="kserve-container"
Sep 16 19:32:00.582448 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.582432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.585227 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585200    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:32:00.585227 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585203    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:00.585442 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:00.585442 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:32:00.585442 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nwxct\""
Sep 16 19:32:00.585442 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.585431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a291d0-predictor-serving-cert\""
Sep 16 19:32:00.586348 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.586327    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a291d0-kube-rbac-proxy-sar-config\""
Sep 16 19:32:00.594485 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.594444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:32:00.617129 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.617097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.617242 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.617134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.617242 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.617161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbcq\" (UniqueName: \"kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.617340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.617261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.617340 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.617315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718005 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.717971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718159 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718159 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbcq\" (UniqueName: \"kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718159 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718330 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718521 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718791 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.718899 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.718848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.720744 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.720724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.728538 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.728516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbcq\" (UniqueName: \"kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq\") pod \"isvc-primary-a291d0-predictor-7f984c747b-z922b\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") " pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:00.892761 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:00.892678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:01.168366 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:01.168341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:32:01.171058 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:32:01.171029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a204f61_f2bc_4f0c_9808_829b9c8f3c50.slice/crio-daa6247d74badc63e9af76ea91ff13084bf33087491023e19e06ab73ea9be0f5 WatchSource:0}: Error finding container daa6247d74badc63e9af76ea91ff13084bf33087491023e19e06ab73ea9be0f5: Status 404 returned error can't find the container with id daa6247d74badc63e9af76ea91ff13084bf33087491023e19e06ab73ea9be0f5
Sep 16 19:32:02.061023 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:02.060988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerStarted","Data":"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"}
Sep 16 19:32:02.061401 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:02.061068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerStarted","Data":"daa6247d74badc63e9af76ea91ff13084bf33087491023e19e06ab73ea9be0f5"}
Sep 16 19:32:06.078748 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:06.078713    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerID="681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292" exitCode=0
Sep 16 19:32:06.079151 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:06.078786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerDied","Data":"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"}
Sep 16 19:32:07.085647 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.085615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerStarted","Data":"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"}
Sep 16 19:32:07.086046 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.085676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerStarted","Data":"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"}
Sep 16 19:32:07.086046 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.085957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:07.086168 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.086114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:07.087286 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.087263    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:07.108092 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:07.108041    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podStartSLOduration=3.19884293 podStartE2EDuration="7.108028681s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.909185751s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:07.106292894 +0000 UTC m=+3110.825390005" watchObservedRunningTime="2026-09-16 19:32:07.108028681 +0000 UTC m=+3110.827125758"
Sep 16 19:32:08.090335 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:08.090296    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:13.094822 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:13.094788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:32:13.095378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:13.095347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:23.096002 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:23.095962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:33.095722 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:33.095684    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:43.095503 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:43.095395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:32:53.095696 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:32:53.095656    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:33:03.095757 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:03.095716    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:33:13.095443 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:13.095393    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:33:17.744403 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:17.744376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:33:35.744265 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.744236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:35.747672 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.747653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:35.750275 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.750252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a291d0\""
Sep 16 19:33:35.750378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.750253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a291d0-dockercfg-s7tw9\""
Sep 16 19:33:35.750378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.750286    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a291d0-kube-rbac-proxy-sar-config\""
Sep 16 19:33:35.750378 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.750344    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a291d0-predictor-serving-cert\""
Sep 16 19:33:35.759601 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.759582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:35.909666 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.909623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:35.909666 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.909661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:35.909868 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.909735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:35.909868 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.909829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:35.909868 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:35.909862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010406 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010406 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010406 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010718 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010718 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.010718 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:36.010508    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-serving-cert: secret "isvc-secondary-a291d0-predictor-serving-cert" not found
Sep 16 19:33:36.010718 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:36.010638    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls podName:e7c4cbf9-03b0-4a31-87e3-e0eee6661f37 nodeName:}" failed. No retries permitted until 2026-09-16 19:33:36.510603957 +0000 UTC m=+3200.229701027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls") pod "isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37") : secret "isvc-secondary-a291d0-predictor-serving-cert" not found
Sep 16 19:33:36.010930 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.010891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.011042 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.011023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.011079 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.011066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.021483 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.020060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.515306 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.515274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.517797 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.517767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") pod \"isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") " pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.658394 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.658353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:36.934741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:36.934716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:36.937401 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:33:36.937375    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c4cbf9_03b0_4a31_87e3_e0eee6661f37.slice/crio-275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c WatchSource:0}: Error finding container 275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c: Status 404 returned error can't find the container with id 275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c
Sep 16 19:33:37.486879 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:37.486840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerStarted","Data":"d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"}
Sep 16 19:33:37.487088 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:37.486906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerStarted","Data":"275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c"}
Sep 16 19:33:40.502004 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:40.501979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/0.log"
Sep 16 19:33:40.502357 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:40.502014    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerID="d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c" exitCode=1
Sep 16 19:33:40.502357 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:40.502092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerDied","Data":"d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"}
Sep 16 19:33:41.508626 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:41.508599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/0.log"
Sep 16 19:33:41.508997 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:41.508715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerStarted","Data":"1c8d6151d0cbd618de89f16035469e2a578028f7b2747e2cae6b9e72f721aa73"}
Sep 16 19:33:44.528054 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.527979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/1.log"
Sep 16 19:33:44.528542 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.528446    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/0.log"
Sep 16 19:33:44.528542 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.528523    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerID="1c8d6151d0cbd618de89f16035469e2a578028f7b2747e2cae6b9e72f721aa73" exitCode=1
Sep 16 19:33:44.528671 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.528599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerDied","Data":"1c8d6151d0cbd618de89f16035469e2a578028f7b2747e2cae6b9e72f721aa73"}
Sep 16 19:33:44.528726 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.528673    2579 scope.go:117] "RemoveContainer" containerID="d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"
Sep 16 19:33:44.529023 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.529000    2579 scope.go:117] "RemoveContainer" containerID="d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"
Sep 16 19:33:44.539726 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:44.539702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_kserve-ci-e2e-test(e7c4cbf9-03b0-4a31-87e3-e0eee6661f37)\"" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"
Sep 16 19:33:44.539990 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:44.539963    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_kserve-ci-e2e-test_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37_0 in pod sandbox 275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c from index: no such id: 'd395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c'" containerID="d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"
Sep 16 19:33:44.540060 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:44.539995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_kserve-ci-e2e-test_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37_0 in pod sandbox 275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c from index: no such id: 'd395ce9db7fc3c9b3b13fe37736becbfa658a8fe165561b47f6c0bb4d5f1420c'"
Sep 16 19:33:45.534806 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:45.534775    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/1.log"
Sep 16 19:33:51.779655 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.779624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:51.809101 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.809063    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:33:51.809382 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.809349    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container" containerID="cri-o://f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6" gracePeriod=30
Sep 16 19:33:51.809535 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.809373    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kube-rbac-proxy" containerID="cri-o://c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c" gracePeriod=30
Sep 16 19:33:51.933779 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.933758    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/1.log"
Sep 16 19:33:51.935898 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:51.935882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:52.033634 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k\") pod \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") "
Sep 16 19:33:52.033634 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location\") pod \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") "
Sep 16 19:33:52.033818 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") pod \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") "
Sep 16 19:33:52.033818 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert\") pod \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") "
Sep 16 19:33:52.033818 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config\") pod \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\" (UID: \"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37\") "
Sep 16 19:33:52.033966 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.033941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:52.034026 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.034003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:52.034120 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.034096    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:52.035856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.035833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:52.035986 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.035968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k" (OuterVolumeSpecName: "kube-api-access-qqm9k") pod "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" (UID: "e7c4cbf9-03b0-4a31-87e3-e0eee6661f37"). InnerVolumeSpecName "kube-api-access-qqm9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:52.134863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.134825    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqm9k\" (UniqueName: \"kubernetes.io/projected/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-api-access-qqm9k\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:52.134863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.134859    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:52.134863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.134870    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:52.134863 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.134879    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:52.135091 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.134886    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:52.567323 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.567290    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerID="c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c" exitCode=2
Sep 16 19:33:52.567539 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.567362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerDied","Data":"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"}
Sep 16 19:33:52.568544 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.568530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp_e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/storage-initializer/1.log"
Sep 16 19:33:52.568628 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.568619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp" event={"ID":"e7c4cbf9-03b0-4a31-87e3-e0eee6661f37","Type":"ContainerDied","Data":"275e478d7242a5a39f080297f993a695fded225bdf88496d4f67369eb2a9de6c"}
Sep 16 19:33:52.568628 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.568622    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"
Sep 16 19:33:52.568741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.568666    2579 scope.go:117] "RemoveContainer" containerID="1c8d6151d0cbd618de89f16035469e2a578028f7b2747e2cae6b9e72f721aa73"
Sep 16 19:33:52.607535 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.607497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:52.611870 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.611837    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a291d0-predictor-64fbbf99f5-tvxcp"]
Sep 16 19:33:52.748615 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:52.748584    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" path="/var/lib/kubelet/pods/e7c4cbf9-03b0-4a31-87e3-e0eee6661f37/volumes"
Sep 16 19:33:53.091469 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:53.091422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 16 19:33:56.252507 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.252479    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:33:56.269320 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxbcq\" (UniqueName: \"kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq\") pod \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") "
Sep 16 19:33:56.269491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config\") pod \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") "
Sep 16 19:33:56.269491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls\") pod \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") "
Sep 16 19:33:56.269491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert\") pod \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") "
Sep 16 19:33:56.269491 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location\") pod \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\" (UID: \"1a204f61-f2bc-4f0c-9808-829b9c8f3c50\") "
Sep 16 19:33:56.269784 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a204f61-f2bc-4f0c-9808-829b9c8f3c50" (UID: "1a204f61-f2bc-4f0c-9808-829b9c8f3c50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:56.269901 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a204f61-f2bc-4f0c-9808-829b9c8f3c50" (UID: "1a204f61-f2bc-4f0c-9808-829b9c8f3c50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:56.269969 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.269898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a204f61-f2bc-4f0c-9808-829b9c8f3c50" (UID: "1a204f61-f2bc-4f0c-9808-829b9c8f3c50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:56.271560 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.271538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq" (OuterVolumeSpecName: "kube-api-access-dxbcq") pod "1a204f61-f2bc-4f0c-9808-829b9c8f3c50" (UID: "1a204f61-f2bc-4f0c-9808-829b9c8f3c50"). InnerVolumeSpecName "kube-api-access-dxbcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:56.271652 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.271632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a204f61-f2bc-4f0c-9808-829b9c8f3c50" (UID: "1a204f61-f2bc-4f0c-9808-829b9c8f3c50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:56.370905 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.370875    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbcq\" (UniqueName: \"kubernetes.io/projected/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-api-access-dxbcq\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.370905 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.370904    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.371078 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.370918    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-proxy-tls\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.371078 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.370931    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-cabundle-cert\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.371078 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.370941    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a204f61-f2bc-4f0c-9808-829b9c8f3c50-kserve-provision-location\") on node \"ip-10-0-136-81.ec2.internal\" DevicePath \"\""
Sep 16 19:33:56.590236 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.590200    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerID="f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6" exitCode=0
Sep 16 19:33:56.590236 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.590239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerDied","Data":"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"}
Sep 16 19:33:56.590504 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.590262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"
Sep 16 19:33:56.590504 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.590309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b" event={"ID":"1a204f61-f2bc-4f0c-9808-829b9c8f3c50","Type":"ContainerDied","Data":"daa6247d74badc63e9af76ea91ff13084bf33087491023e19e06ab73ea9be0f5"}
Sep 16 19:33:56.590504 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.590336    2579 scope.go:117] "RemoveContainer" containerID="c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"
Sep 16 19:33:56.599780 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.599761    2579 scope.go:117] "RemoveContainer" containerID="f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"
Sep 16 19:33:56.607414 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.607386    2579 scope.go:117] "RemoveContainer" containerID="681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"
Sep 16 19:33:56.615312 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615295    2579 scope.go:117] "RemoveContainer" containerID="c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"
Sep 16 19:33:56.615528 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:33:56.615589 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:56.615574    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c\": container with ID starting with c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c not found: ID does not exist" containerID="c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"
Sep 16 19:33:56.615628 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615599    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c"} err="failed to get container status \"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c\": rpc error: code = NotFound desc = could not find container \"c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c\": container with ID starting with c486f26f2b4accbdfb0d580425374e6f6dcce6be22f50464ec6bb7dc4fcf729c not found: ID does not exist"
Sep 16 19:33:56.615628 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615619    2579 scope.go:117] "RemoveContainer" containerID="f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"
Sep 16 19:33:56.615894 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:56.615871    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6\": container with ID starting with f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6 not found: ID does not exist" containerID="f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"
Sep 16 19:33:56.615949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6"} err="failed to get container status \"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6\": rpc error: code = NotFound desc = could not find container \"f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6\": container with ID starting with f190156803f50def9e461ae8ffa30427660d988897c663d4874b80c2adaf2be6 not found: ID does not exist"
Sep 16 19:33:56.615949 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.615918    2579 scope.go:117] "RemoveContainer" containerID="681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"
Sep 16 19:33:56.616148 ip-10-0-136-81 kubenswrapper[2579]: E0916 19:33:56.616134    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292\": container with ID starting with 681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292 not found: ID does not exist" containerID="681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"
Sep 16 19:33:56.616189 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.616153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292"} err="failed to get container status \"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292\": rpc error: code = NotFound desc = could not find container \"681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292\": container with ID starting with 681e091c77d59ed827ab7f2f55322446fecdaa30f47274dbad262040e0a94292 not found: ID does not exist"
Sep 16 19:33:56.622061 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.622015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a291d0-predictor-7f984c747b-z922b"]
Sep 16 19:33:56.748917 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:33:56.748888    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" path="/var/lib/kubelet/pods/1a204f61-f2bc-4f0c-9808-829b9c8f3c50/volumes"
Sep 16 19:34:53.214853 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.214824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vps5/must-gather-z94h6"]
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215146    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kube-rbac-proxy"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kube-rbac-proxy"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215172    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215177    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215186    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215191    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215198    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215203    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215210    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215215    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215270    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215279    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kube-rbac-proxy"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215285    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c4cbf9-03b0-4a31-87e3-e0eee6661f37" containerName="storage-initializer"
Sep 16 19:34:53.215294 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.215292    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a204f61-f2bc-4f0c-9808-829b9c8f3c50" containerName="kserve-container"
Sep 16 19:34:53.218255 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.218241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.220921 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.220898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vps5\"/\"kube-root-ca.crt\""
Sep 16 19:34:53.222149 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.222133    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vps5\"/\"openshift-service-ca.crt\""
Sep 16 19:34:53.222220 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.222137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vps5\"/\"default-dockercfg-j62vv\""
Sep 16 19:34:53.225757 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.225738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/must-gather-z94h6"]
Sep 16 19:34:53.324573 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.324546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6a9d803-240e-4dd4-bf78-182e969e802f-must-gather-output\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.324682 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.324583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z826\" (UniqueName: \"kubernetes.io/projected/d6a9d803-240e-4dd4-bf78-182e969e802f-kube-api-access-6z826\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.425751 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.425727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6a9d803-240e-4dd4-bf78-182e969e802f-must-gather-output\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.425843 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.425758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z826\" (UniqueName: \"kubernetes.io/projected/d6a9d803-240e-4dd4-bf78-182e969e802f-kube-api-access-6z826\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.426057 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.426040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6a9d803-240e-4dd4-bf78-182e969e802f-must-gather-output\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.434974 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.434952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z826\" (UniqueName: \"kubernetes.io/projected/d6a9d803-240e-4dd4-bf78-182e969e802f-kube-api-access-6z826\") pod \"must-gather-z94h6\" (UID: \"d6a9d803-240e-4dd4-bf78-182e969e802f\") " pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.541977 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.541915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/must-gather-z94h6"
Sep 16 19:34:53.811113 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.811090    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/must-gather-z94h6"]
Sep 16 19:34:53.813869 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:34:53.813840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a9d803_240e_4dd4_bf78_182e969e802f.slice/crio-f8e4392d775a29c949988013a938419911eccc72bcfdf5fd0f5afd8fee0011da WatchSource:0}: Error finding container f8e4392d775a29c949988013a938419911eccc72bcfdf5fd0f5afd8fee0011da: Status 404 returned error can't find the container with id f8e4392d775a29c949988013a938419911eccc72bcfdf5fd0f5afd8fee0011da
Sep 16 19:34:53.815760 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.815743    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:34:53.832671 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:53.832646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/must-gather-z94h6" event={"ID":"d6a9d803-240e-4dd4-bf78-182e969e802f","Type":"ContainerStarted","Data":"f8e4392d775a29c949988013a938419911eccc72bcfdf5fd0f5afd8fee0011da"}
Sep 16 19:34:54.840141 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:54.840093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/must-gather-z94h6" event={"ID":"d6a9d803-240e-4dd4-bf78-182e969e802f","Type":"ContainerStarted","Data":"ce50f4c42dc32a8aaf9ba9b829def371cb9d0caabe895dd33086f05ec92c75bf"}
Sep 16 19:34:54.840585 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:54.840159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/must-gather-z94h6" event={"ID":"d6a9d803-240e-4dd4-bf78-182e969e802f","Type":"ContainerStarted","Data":"24bd2ece79c5cc2d339c2d27acd7b08395498e09f3cd3417492ce7d315db37e7"}
Sep 16 19:34:54.856954 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:54.856907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vps5/must-gather-z94h6" podStartSLOduration=1.222515337 podStartE2EDuration="1.856890744s" totalImagesPullingTime="634.375407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:54.85521114 +0000 UTC m=+3278.574308219" watchObservedRunningTime="2026-09-16 19:34:54.856890744 +0000 UTC m=+3278.575987842"
Sep 16 19:34:56.154947 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:56.154917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h69vp_ab1836ce-5a61-4d27-b9fc-7ca2c7d36cf2/global-pull-secret-syncer/0.log"
Sep 16 19:34:56.287135 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:56.287102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d65wk_c9f51084-955f-4058-b1e2-2da5bcea710b/konnectivity-agent/0.log"
Sep 16 19:34:56.342547 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:34:56.342486    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-81.ec2.internal_2d64174e840dde57cd70ce29f4e8fd34/haproxy/0.log"
Sep 16 19:35:00.730000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.729964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/alertmanager/0.log"
Sep 16 19:35:00.762848 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.762818    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/config-reloader/0.log"
Sep 16 19:35:00.790073 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.790047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/kube-rbac-proxy-web/0.log"
Sep 16 19:35:00.813589 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.813560    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/kube-rbac-proxy/0.log"
Sep 16 19:35:00.840040 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.839997    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/kube-rbac-proxy-metric/0.log"
Sep 16 19:35:00.868319 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.868289    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/prom-label-proxy/0.log"
Sep 16 19:35:00.924623 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:00.924587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_826d7ec7-6587-43c4-9279-4d97ccac5b85/init-config-reloader/0.log"
Sep 16 19:35:01.005527 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.005431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-g8krb_9492d0dd-e557-484f-80e5-a21721236846/kube-state-metrics/0.log"
Sep 16 19:35:01.032380 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.032351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-g8krb_9492d0dd-e557-484f-80e5-a21721236846/kube-rbac-proxy-main/0.log"
Sep 16 19:35:01.057310 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.057284    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-g8krb_9492d0dd-e557-484f-80e5-a21721236846/kube-rbac-proxy-self/0.log"
Sep 16 19:35:01.248851 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.248824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj85s_f2edca00-829c-4dc5-8731-99bce57a60d3/node-exporter/0.log"
Sep 16 19:35:01.273901 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.273832    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj85s_f2edca00-829c-4dc5-8731-99bce57a60d3/kube-rbac-proxy/0.log"
Sep 16 19:35:01.293542 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.293514    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pj85s_f2edca00-829c-4dc5-8731-99bce57a60d3/init-textfile/0.log"
Sep 16 19:35:01.938960 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.938928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/thanos-query/0.log"
Sep 16 19:35:01.962795 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.962756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/kube-rbac-proxy-web/0.log"
Sep 16 19:35:01.990749 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:01.990629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/kube-rbac-proxy/0.log"
Sep 16 19:35:02.016087 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:02.016054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/prom-label-proxy/0.log"
Sep 16 19:35:02.043311 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:02.043270    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/kube-rbac-proxy-rules/0.log"
Sep 16 19:35:02.070111 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:02.070061    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffcfc849c-ghbc9_b328323d-de90-4291-bb87-8011bba6dbaf/kube-rbac-proxy-metrics/0.log"
Sep 16 19:35:03.102914 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.102874    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-qvssf_90af4f64-52de-4dd6-8ce7-7cfb0022fb05/networking-console-plugin/0.log"
Sep 16 19:35:03.828143 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.828099    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"]
Sep 16 19:35:03.831952 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.831927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:03.849481 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.849427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"]
Sep 16 19:35:03.925322 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.925291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-lib-modules\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:03.925524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.925333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2f4\" (UniqueName: \"kubernetes.io/projected/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-kube-api-access-jm2f4\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:03.925524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.925360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-podres\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:03.925524 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.925449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-proc\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:03.925652 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:03.925576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-sys\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.025973 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.025935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-lib-modules\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.025973 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.025975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2f4\" (UniqueName: \"kubernetes.io/projected/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-kube-api-access-jm2f4\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.025992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-podres\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-proc\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-sys\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-lib-modules\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-proc\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-sys\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.026217 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.026182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-podres\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.035045 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.035022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2f4\" (UniqueName: \"kubernetes.io/projected/17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5-kube-api-access-jm2f4\") pod \"perf-node-gather-daemonset-nx4sl\" (UID: \"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5\") " pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.040483 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.040443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-cw9ft_015dd585-fed3-4d0b-b652-64f27f0d000c/download-server/0.log"
Sep 16 19:35:04.144513 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.144417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.423603 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.423522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"]
Sep 16 19:35:04.426704 ip-10-0-136-81 kubenswrapper[2579]: W0916 19:35:04.426672    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17fe8ce3_9deb_4b7b_92e8_eebbbd3298c5.slice/crio-34b875b7fe036f744ded47dab25f4eb2c947289096c2ff33e03778188dd4d68f WatchSource:0}: Error finding container 34b875b7fe036f744ded47dab25f4eb2c947289096c2ff33e03778188dd4d68f: Status 404 returned error can't find the container with id 34b875b7fe036f744ded47dab25f4eb2c947289096c2ff33e03778188dd4d68f
Sep 16 19:35:04.597394 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.597363    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-pppw2_c95fdaad-b588-4818-b2eb-1ca0eece2d31/volume-data-source-validator/0.log"
Sep 16 19:35:04.925636 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.925572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl" event={"ID":"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5","Type":"ContainerStarted","Data":"72a3072f60c7026a91d6efa17d98d1ee015349ce440f19d32a0235f9b7e84a33"}
Sep 16 19:35:04.925856 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.925681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl" event={"ID":"17fe8ce3-9deb-4b7b-92e8-eebbbd3298c5","Type":"ContainerStarted","Data":"34b875b7fe036f744ded47dab25f4eb2c947289096c2ff33e03778188dd4d68f"}
Sep 16 19:35:04.926470 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.926422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:04.951162 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:04.951098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl" podStartSLOduration=1.951078504 podStartE2EDuration="1.951078504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:04.948204695 +0000 UTC m=+3288.667301772" watchObservedRunningTime="2026-09-16 19:35:04.951078504 +0000 UTC m=+3288.670175582"
Sep 16 19:35:05.757902 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:05.757873    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvvb2_c77b5c39-4844-45ed-afcf-779b2f3b0145/dns/0.log"
Sep 16 19:35:05.779839 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:05.779815    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvvb2_c77b5c39-4844-45ed-afcf-779b2f3b0145/kube-rbac-proxy/0.log"
Sep 16 19:35:05.807048 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:05.807010    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mh7lg_4a2385e9-c1ed-48e9-91fd-d3a66af674b3/dns-node-resolver/0.log"
Sep 16 19:35:06.338067 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:06.338017    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f894ffccc-qzt2k_90ab2308-69cd-46b1-98e9-bfea1334b3f0/registry/0.log"
Sep 16 19:35:06.366733 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:06.366709    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fjkkt_786b0e59-4b3f-47c7-9c97-cda697025943/node-ca/0.log"
Sep 16 19:35:07.539329 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:07.539301    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn88h_ccc6de4b-def5-4625-b397-0791b85b12b2/serve-healthcheck-canary/0.log"
Sep 16 19:35:07.929764 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:07.929732    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qsnj6_ff45e963-083a-410b-a861-7f5379c427ac/insights-operator/0.log"
Sep 16 19:35:07.930000 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:07.929983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qsnj6_ff45e963-083a-410b-a861-7f5379c427ac/insights-operator/1.log"
Sep 16 19:35:08.034196 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:08.034173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2tjk_7d82e923-85db-4b4f-bc3c-9991af8ee5cd/kube-rbac-proxy/0.log"
Sep 16 19:35:08.061114 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:08.061091    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2tjk_7d82e923-85db-4b4f-bc3c-9991af8ee5cd/exporter/0.log"
Sep 16 19:35:08.089752 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:08.089727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2tjk_7d82e923-85db-4b4f-bc3c-9991af8ee5cd/extractor/0.log"
Sep 16 19:35:10.611016 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:10.610983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-557b59fdc4-rf7m9_3b1b4d00-04a5-4d8a-b590-095ad3ea74b6/manager/0.log"
Sep 16 19:35:10.667670 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:10.667637    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-nw8vm_3e474aa1-a3d3-4a22-96a3-4442ab26d624/server/0.log"
Sep 16 19:35:10.968619 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:10.968550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-27t59_aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed/s3-init/0.log"
Sep 16 19:35:10.988282 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:10.988257    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-27t59_aaf286e2-8bc4-4e6b-bac5-faac0d3bdbed/download-hf-model/0.log"
Sep 16 19:35:11.021558 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.021530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-9blhl_f4e26eff-2d34-4c75-b192-3666e2b9c1a3/s3-tls-init-custom/0.log"
Sep 16 19:35:11.041439 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.041414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-9blhl_f4e26eff-2d34-4c75-b192-3666e2b9c1a3/download-hf-model/0.log"
Sep 16 19:35:11.075007 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.074980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-smn88_7a07856c-fd6a-400c-b6fc-abce7655cd18/s3-tls-init-serving/0.log"
Sep 16 19:35:11.093596 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.093573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-smn88_7a07856c-fd6a-400c-b6fc-abce7655cd18/download-hf-model/0.log"
Sep 16 19:35:11.156789 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.156758    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-lmzgd_aff80783-128f-4457-9323-868d469118b0/seaweedfs-tls-custom/0.log"
Sep 16 19:35:11.947268 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:11.947242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vps5/perf-node-gather-daemonset-nx4sl"
Sep 16 19:35:16.106549 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:16.106521    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lqzxm_d0db4211-6f85-4e65-9cb7-bd14ae7f782f/kube-storage-version-migrator-operator/1.log"
Sep 16 19:35:16.107861 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:16.107841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-lqzxm_d0db4211-6f85-4e65-9cb7-bd14ae7f782f/kube-storage-version-migrator-operator/0.log"
Sep 16 19:35:17.059827 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.059797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4chnh_932754fd-04f5-4fc3-bd81-36b598b7311a/kube-multus/0.log"
Sep 16 19:35:17.611901 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.611873    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:35:17.640028 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.639998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/egress-router-binary-copy/0.log"
Sep 16 19:35:17.673867 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.673844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/cni-plugins/0.log"
Sep 16 19:35:17.701219 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.701198    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/bond-cni-plugin/0.log"
Sep 16 19:35:17.726568 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.726547    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/routeoverride-cni/0.log"
Sep 16 19:35:17.810251 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.810217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/whereabouts-cni-bincopy/0.log"
Sep 16 19:35:17.830370 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.830348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xmplp_eb3c189b-2662-4686-accc-a20201cd7ef3/whereabouts-cni/0.log"
Sep 16 19:35:17.930741 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.930650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bc7ds_7633ccec-856e-413c-8d65-3aaeb1201b6a/network-metrics-daemon/0.log"
Sep 16 19:35:17.954680 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:17.954659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bc7ds_7633ccec-856e-413c-8d65-3aaeb1201b6a/kube-rbac-proxy/0.log"
Sep 16 19:35:19.348225 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.348195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-controller/0.log"
Sep 16 19:35:19.366045 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.366018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/1.log"
Sep 16 19:35:19.366045 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.366035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovn-acl-logging/0.log"
Sep 16 19:35:19.385297 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.385277    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/kube-rbac-proxy-node/0.log"
Sep 16 19:35:19.408432 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.408407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:35:19.428804 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.428759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/northd/0.log"
Sep 16 19:35:19.451907 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.451890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/nbdb/0.log"
Sep 16 19:35:19.477740 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.477687    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/sbdb/0.log"
Sep 16 19:35:19.531428 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:19.531402    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tn8vm_93793e48-633c-432e-b5e7-8eeba7006e71/ovnkube-controller/0.log"
Sep 16 19:35:20.524749 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:20.524725    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cdlgj_1c88996b-5555-47e0-91a8-618176e2d3d7/network-check-target-container/0.log"
Sep 16 19:35:21.547150 ip-10-0-136-81 kubenswrapper[2579]: I0916 19:35:21.547117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6p5cm_5b44a16f-1b5d-46db-a58f-1ac32c313498/iptables-alerter/0.log"