Sep 09 19:53:12.749600 ip-10-0-135-16 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 19:53:12.749611 ip-10-0-135-16 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 19:53:12.749618 ip-10-0-135-16 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 19:53:12.749770 ip-10-0-135-16 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 19:53:22.784683 ip-10-0-135-16 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 19:53:22.784701 ip-10-0-135-16 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 18850472d0734291a53dc10ec9671224 --
Sep 09 19:55:10.495002 ip-10-0-135-16 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:55:10.915714 ip-10-0-135-16 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:10.915714 ip-10-0-135-16 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:55:10.915714 ip-10-0-135-16 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:10.915714 ip-10-0-135-16 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:55:10.915714 ip-10-0-135-16 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:10.916826 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.916739    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:55:10.922644 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922629    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.922644 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922644    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922647    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922651    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922654    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922657    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922662    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922665    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922671    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922674    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922687    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922690    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922693    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922696    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922699    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922701    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922704    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922706    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922709    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922713    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922715    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.922718 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922718    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922720    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922723    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922726    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922729    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922731    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922734    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922737    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922739    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922742    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922745    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922747    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922750    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922752    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922755    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922757    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922760    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922763    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922765    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922768    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.923192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922770    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922779    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922781    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922784    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922786    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922790    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922792    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922795    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922797    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922800    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922802    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922805    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922807    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922810    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922814    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922817    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922819    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922822    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922825    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922827    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.923672 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922830    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922832    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922835    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922837    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922840    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922843    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922845    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922848    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922850    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922852    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922855    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922857    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922860    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922863    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922865    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922868    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922872    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922876    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922880    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.924228 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922883    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922886    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922888    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922893    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922896    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922898    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922901    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922904    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922907    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922909    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922913    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922916    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922919    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922923    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922927    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922931    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922934    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922936    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.922939    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.924711 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923631    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923638    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923641    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923644    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923647    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923650    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923653    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923655    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923658    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923661    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923664    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923667    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923669    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923673    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923676    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923692    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923695    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923698    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923701    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923703    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.925170 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923706    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923709    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923711    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923714    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923716    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923720    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923724    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923727    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923730    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923733    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923736    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923738    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923741    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923743    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923746    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923748    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923752    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923754    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923757    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.925751 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923759    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923762    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923764    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923767    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923770    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923772    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923775    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923778    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923780    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923783    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923785    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923788    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923791    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923794    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923797    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923799    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923801    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923804    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923807    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923809    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.926237 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923811    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923814    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923816    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923820    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923824    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923826    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923829    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923832    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923835    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923838    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923841    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923843    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923846    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923849    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923852    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923857    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923861    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923864    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923867    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.926731 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923870    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923873    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923876    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923879    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923882    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923885    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923888    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923891    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923893    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923896    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923899    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923901    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923904    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923906    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923909    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923911    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923914    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923916    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923919    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923921    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.927192 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.923924    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924310    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924316    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924319    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924322    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924325    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924329    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924333    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924339    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924342    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924344    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924347    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924350    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924353    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924355    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924358    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924361    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924364    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924367    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924369    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.927685 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924372    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924375    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924379    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924382    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924385    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924388    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924391    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924394    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924397    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924400    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924402    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924405    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924408    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924410    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924413    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924415    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924418    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924420    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924423    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.928184 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924425    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924429    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924431    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924434    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924436    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924439    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924441    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924444    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924447    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924450    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924453    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924456    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924458    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924461    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924464    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924466    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924470    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924473    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924476    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.928647 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924479    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924481    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924484    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924486    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924489    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924492    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924494    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924496    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924499    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924501    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924504    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924507    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924509    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924513    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924518    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924521    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924523    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924526    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924528    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924531    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.929151 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924534    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924537    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924540    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924542    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924545    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924547    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924550    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924552    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924555    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924557    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924560    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924562    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924565    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924567    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924570    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924572    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924574    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924577    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924579    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924582    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.929637 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924584    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.924587    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924649    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924695    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924701    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924707    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924712    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924716    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924720    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924724    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924728    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924730    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924734    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924737    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924742    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924745    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924748    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924751    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924754    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924757    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924761    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924764    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924767    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924770    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:55:10.930130 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924773    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924777    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924780    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924783    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924787    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924790    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924793    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924796    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924799    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924802    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924806    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924809    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924812    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924816    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924820    2582 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924823    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924828    2582 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924831    2582 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924834    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924837    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924840    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924843    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924846    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924850    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924853    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:55:10.930722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924855    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924859    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924862    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924864    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924867    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924870    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924873    2582 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924877    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924880    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924883    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924886    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924889    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924891    2582 flags.go:64] FLAG: --help="false"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924895    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924898    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924901    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924904    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924907    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924910    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924913    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924917    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924922    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924924    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:55:10.931317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924927    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924930    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924933    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924936    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924939    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924942    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924945    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924948    2582 flags.go:64] FLAG: --lock-file=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924951    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924954    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924957    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924962    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924965    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924967    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924970    2582 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924973    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924977    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924979    2582 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924982    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924986    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924989    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924993    2582 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924995    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.924998    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925001    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:55:10.931898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925004    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925007    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925010    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925013    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925022    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925026    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925029    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925032    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925035    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925041    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925044    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925047    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925050    2582 flags.go:64] FLAG: --port="10250"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925053    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925056    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cad745e156feab9a"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925059    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925062    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925065    2582 flags.go:64] FLAG: --register-node="true"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925068    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925075    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925077    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925080    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925083    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925086    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925089    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:55:10.932887 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925092    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925095    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925098    2582 flags.go:64] FLAG: --runonce="false"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925101    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925104    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925107    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925110    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925113    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925116    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925119    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925122    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925127    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925130    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925134    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925136    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925140    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925143    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925146    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925152    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925155    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925158    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925162    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925165    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925168    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925171    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:55:10.933960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925174    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925177    2582 flags.go:64] FLAG: --v="2"
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925181    2582 flags.go:64] FLAG: --version="false"
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925185    2582 flags.go:64] FLAG: --vmodule=""
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925189    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.925192    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925303    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925307    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925310    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925313    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925315    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925318    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925321    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925323    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925326    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925328    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925331    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925334    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925338    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925340    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925344    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.934629 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925346    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925349    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925352    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925354    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925357    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925359    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925362    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925365    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925367    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925370    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925372    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925390    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925393    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925396    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925399    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925402    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925406    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925408    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925411    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925413    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.935174 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925416    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925419    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925422    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925425    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925428    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925430    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925433    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925436    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925438    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925442    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925445    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925447    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925450    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925453    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925455    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925458    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925460    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925463    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925466    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925468    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.935659 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925471    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925475    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925478    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925481    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925485    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925488    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925490    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925493    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925495    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925498    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925500    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925503    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925506    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925508    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925511    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925514    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925516    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925519    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925521    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.936173 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925524    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925526    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925530    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925533    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925536    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925538    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925541    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925543    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925546    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925548    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925552    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925555    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925558    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925560    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925563    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925565    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925569    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925573    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925576    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.936653 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925579    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925582    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925585    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925587    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925590    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.925593    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.926288    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.935143    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.935158    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.935975    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.935986    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936045    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936051    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936055    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936059    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.937178 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936062    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936066    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936070    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936073    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936076    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936078    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936081    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936083    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936086    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936089    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936091    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936094    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936097    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936100    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936103    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936107    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936109    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936112    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936115    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936118    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.937549 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936120    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936123    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936126    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936128    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936138    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936142    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936145    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936148    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936151    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936154    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936156    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936159    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936162    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936164    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936167    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936169    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936172    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936175    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936179    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936182    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.938050 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936186    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936189    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936193    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936196    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936199    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936202    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936205    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936208    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936210    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936213    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936215    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936218    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936220    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936224    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936226    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936229    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936233    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936236    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936239    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.938536 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936242    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936244    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936248    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936250    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936253    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936256    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936259    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936262    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936265    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936267    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936270    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936272    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936275    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936278    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936280    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936283    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936286    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936288    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936291    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936293    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.939021 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936296    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936298    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936301    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936304    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936306    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936309    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936311    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936314    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936317    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936319    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936322    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936324    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936327    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936329    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936332    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.939495 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936334    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.936339    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936457    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936463    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936466    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936469    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936472    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936475    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936478    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936481    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936483    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936486    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936489    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936491    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:10.939875 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936494    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936497    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936500    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936502    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936505    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936507    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936510    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936512    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936515    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936517    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936520    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936523    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936525    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936528    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936530    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936533    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936535    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936538    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936541    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936543    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:10.940243 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936546    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936548    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936551    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936554    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936557    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936559    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936562    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936564    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936567    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936569    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936572    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936574    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936577    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936580    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936583    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936585    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936588    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936591    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936593    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936596    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:10.940743 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936598    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936601    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936604    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936606    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936609    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936611    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936614    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936616    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936619    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936621    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936624    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936627    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936629    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936632    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936635    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936638    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936640    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936643    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936645    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936647    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:10.941231 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936651    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936655    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936658    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936661    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936664    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936667    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936670    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936673    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936689    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936694    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936697    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936700    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936703    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936705    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936708    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936711    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936713    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936716    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936719    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936721    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:10.941722 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936724    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936726    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936729    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936732    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936734    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936737    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:10.936740    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.936745    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.936848    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.939198    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.940227    2582 server.go:1019] "Starting client certificate rotation"
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.940323    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:10.942196 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.940354    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:10.943213 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.943199    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:10.946905 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.946892    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:10.957076 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.957058    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:55:10.961085 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.961069    2582 log.go:25] "Validated CRI v1 image API"
Sep 09 19:55:10.961921 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.961902    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:55:10.966412 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.966385    2582 fs.go:135] Filesystem UUIDs: map[6b7a540e-0c29-45af-a4df-ce0070ca90f8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aca1940f-2e73-4e5d-9a71-2cbfdae1785d:/dev/nvme0n1p4]
Sep 09 19:55:10.966462 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.966412    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:55:10.971083 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.971058    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:10.973451 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.973334    2582 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:10.970765666 +0000 UTC m=+0.371442849 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099663 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26f7fffab449a88d94db0a3179cf0e SystemUUID:ec26f7ff-fab4-49a8-8d94-db0a3179cf0e BootID:18850472-d073-4291-a53d-c10ec9671224 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d3:3d:78:89:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d3:3d:78:89:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:d5:2d:63:9e:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:55:10.973451 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.973447    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:55:10.973559 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.973525    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:55:10.977078 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.977057    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:55:10.977266 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.977088    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:55:10.978771 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.978761    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:55:10.978804 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.978774    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:55:10.978804 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.978783    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:10.979585 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.979575    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:10.981894 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.981880    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:55:10.985353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985341    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:10.985476 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985466    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:55:10.985513 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985509    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:55:10.985554 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985521    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:55:10.985554 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985531    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:55:10.985636 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985567    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:55:10.985636 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.985575    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:55:10.988103 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.988084    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:10.988189 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.988113    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:10.992116 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.992097    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:55:10.994881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.994867    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:55:10.994937 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.994897    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:55:10.995867 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995851    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995875    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995885    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995895    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995906    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995912    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995909    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x4tvx"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995927    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:55:10.995938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995936    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:55:10.996160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995943    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:55:10.996160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995954    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:55:10.996160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995967    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:55:10.996160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.995973    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:55:10.996990 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.996980    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:55:10.997032 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:10.996991    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:55:10.998728 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:10.998707    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:55:10.998775 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:10.998706    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:55:11.000908 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.000895    2582 server.go:1268] "Started kubelet"
Sep 09 19:55:11.001010 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.000979    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:55:11.001050 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.001010    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:55:11.001083 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.001072    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:55:11.001785 ip-10-0-135-16 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:55:11.002835 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.002816    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:55:11.004376 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.003455    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x4tvx"
Sep 09 19:55:11.004376 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.003769    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:55:11.007931 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.007902    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:55:11.008878 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.008857    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:55:11.009726 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.009707    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:11.010101 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.008894    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-16.ec2.internal.18d3bf34eb2aa231  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-16.ec2.internal,UID:ip-10-0-135-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-16.ec2.internal,},FirstTimestamp:2026-09-09 19:55:11.000863281 +0000 UTC m=+0.401540483,LastTimestamp:2026-09-09 19:55:11.000863281 +0000 UTC m=+0.401540483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-16.ec2.internal,}"
Sep 09 19:55:11.010507 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.010488    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:55:11.010714 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.010694    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.011284 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.011266    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:55:11.011394 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.011274    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:55:11.011394 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.011393    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:55:11.011574 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.011552    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:55:11.011574 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.011566    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:55:11.012432 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012414    2582 factory.go:55] Registering systemd factory
Sep 09 19:55:11.012432 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012433    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:55:11.012750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012732    2582 factory.go:153] Registering CRI-O factory
Sep 09 19:55:11.012750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012753    2582 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:55:11.012861 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012825    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:55:11.012861 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012839    2582 factory.go:103] Registering Raw factory
Sep 09 19:55:11.012861 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.012848    2582 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:55:11.013391 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.013378    2582 manager.go:349] Starting recovery of all containers
Sep 09 19:55:11.013750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.013733    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:11.017656 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.017627    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-16.ec2.internal\" not found" node="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.026980 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.026797    2582 manager.go:354] Recovery completed
Sep 09 19:55:11.031078 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.031064    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.032794 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.032777    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.032864 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.032809    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.032864 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.032823    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.033359 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.033344    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:55:11.033424 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.033359    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:55:11.033424 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.033377    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:11.033424 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.033350    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.034105 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034088    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.034174 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034115    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.034174 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034125    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.034430 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034420    2582 policy_none.go:49] "None policy: Start"
Sep 09 19:55:11.034460 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034434    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:55:11.034460 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.034443    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:55:11.036155 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.036142    2582 policy_none.go:47] "Start"
Sep 09 19:55:11.073238 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073225    2582 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.073258    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073272    2582 server.go:87] "Starting device plugin registration server"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073570    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073582    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073663    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073743    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.073751    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.074305    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:55:11.083535 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.074344    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.117033 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.117007    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:55:11.118166 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.118142    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:55:11.118246 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.118200    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:55:11.118246 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.118226    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:55:11.118341 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.118269    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:55:11.120992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.120974    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:11.174337 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.174261    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.175392 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.175376    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.175466 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.175409    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.175466 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.175423    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.175466 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.175452    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.184237 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.184220    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.184327 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.184245    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-16.ec2.internal\": node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.218459 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.218440    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"]
Sep 09 19:55:11.218547 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.218516    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.221122 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.221107    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.221201 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.221133    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.221201 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.221155    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.222332 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.222320    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.222454 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.222441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.222487 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.222467    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.224712 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224692    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.224797 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224720    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.224797 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224729    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.224797 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224697    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.224797 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224793    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.224930 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.224807    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.225868 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.225852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.225921 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.225884    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:11.226610 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.226590    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:11.226709 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.226623    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:11.226709 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.226637    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:11.230585 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.230115    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.249334 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.249313    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-16.ec2.internal\" not found" node="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.253663 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.253649    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-16.ec2.internal\" not found" node="ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.313198 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.313170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.313198 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.313197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.313365 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.313215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42b537ed4469da6dbc5578dc2b5d442c-config\") pod \"kube-apiserver-proxy-ip-10-0-135-16.ec2.internal\" (UID: \"42b537ed4469da6dbc5578dc2b5d442c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.330963 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.330937    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.413864 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.413992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.413992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42b537ed4469da6dbc5578dc2b5d442c-config\") pod \"kube-apiserver-proxy-ip-10-0-135-16.ec2.internal\" (UID: \"42b537ed4469da6dbc5578dc2b5d442c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.413992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42b537ed4469da6dbc5578dc2b5d442c-config\") pod \"kube-apiserver-proxy-ip-10-0-135-16.ec2.internal\" (UID: \"42b537ed4469da6dbc5578dc2b5d442c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.413992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.413992 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.413933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5b9edf47ad1b8c3a43ea0ebfaa647f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal\" (UID: \"a5b9edf47ad1b8c3a43ea0ebfaa647f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.432065 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.432015    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.533033 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.533001    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.552177 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.552159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.555907 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.555744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:11.633587 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.633539    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.734378 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.734337    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.834649 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.834616    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.935767 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:11.935738    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:11.940271 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.940225    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:55:11.940409 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.940390    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:11.940451 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:11.940417    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:12.012281 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.012244    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:11 +0000 UTC" deadline="2028-06-22 08:50:06.302797902 +0000 UTC"
Sep 09 19:55:12.012281 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.012276    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15636h54m54.290525331s"
Sep 09 19:55:12.036828 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:12.036799    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:12.093458 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:12.093430    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b537ed4469da6dbc5578dc2b5d442c.slice/crio-c6672fbeff96eaeffcaf9b3a3fc79631868f03ff832f8b768fb7e8eb50553f69 WatchSource:0}: Error finding container c6672fbeff96eaeffcaf9b3a3fc79631868f03ff832f8b768fb7e8eb50553f69: Status 404 returned error can't find the container with id c6672fbeff96eaeffcaf9b3a3fc79631868f03ff832f8b768fb7e8eb50553f69
Sep 09 19:55:12.098419 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.098399    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:55:12.115874 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.115852    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:12.121272 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.121227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal" event={"ID":"42b537ed4469da6dbc5578dc2b5d442c","Type":"ContainerStarted","Data":"c6672fbeff96eaeffcaf9b3a3fc79631868f03ff832f8b768fb7e8eb50553f69"}
Sep 09 19:55:12.137580 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:12.137557    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:12.238649 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:12.238573    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:12.339014 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:12.338983    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:12.409593 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:12.409563    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b9edf47ad1b8c3a43ea0ebfaa647f9.slice/crio-aa83e4cc3fbf8110ad8b1d31d820f03d409e1c4d0d9ff31c7c64bf058b968668 WatchSource:0}: Error finding container aa83e4cc3fbf8110ad8b1d31d820f03d409e1c4d0d9ff31c7c64bf058b968668: Status 404 returned error can't find the container with id aa83e4cc3fbf8110ad8b1d31d820f03d409e1c4d0d9ff31c7c64bf058b968668
Sep 09 19:55:12.439762 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:12.439736    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-16.ec2.internal\" not found"
Sep 09 19:55:12.495307 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.495265    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:12.511567 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.511544    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:12.526652 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.526633    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:12.527657 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.527643    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal"
Sep 09 19:55:12.540993 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.540974    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:12.987603 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.987517    2582 apiserver.go:52] "Watching apiserver"
Sep 09 19:55:12.997881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.997850    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:55:12.998273 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:12.998244    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-l4zfq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4","openshift-multus/multus-additional-cni-plugins-cpkgx","openshift-multus/multus-cb56b","openshift-network-diagnostics/network-check-target-xqdk9","openshift-network-operator/iptables-alerter-smrcb","openshift-ovn-kubernetes/ovnkube-node-hr562","kube-system/global-pull-secret-syncer-wwrln","kube-system/konnectivity-agent-csjvb","kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zg7cm","openshift-dns/node-resolver-cn5pp","openshift-image-registry/node-ca-tsnld","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal"]
Sep 09 19:55:13.000116 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.000085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.001352 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.001330    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.002686 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.002664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.002820 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.002781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:55:13.004206 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.004185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.005580 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.005553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:13.005704 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.005629    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:13.006880 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.006858    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:55:13.007350 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.007098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.008815 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.008513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.009171 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.009145    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.009308 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.009248    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.010068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010046    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:13.010165 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.010165 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.010144    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:13.010528 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:55:13.010647 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010618    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:55:13.010773 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:55:13.010841 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.010796    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mpsrs\""
Sep 09 19:55:13.011584 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.011566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.012953 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.012934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.013041 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.013001    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:13.014227 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.014209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.015567 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.015546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.017972 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.017682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:55:13.019658 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.019634    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.019770 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.019667    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.020305 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.020287    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.020917 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.020709    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.020917 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.020730    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:55:13.021092 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.020985    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.021092 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.021019    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.021092 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.021033    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:55:13.022151 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.022062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:55:13.022151 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.022135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hm5s7\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.022932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.022994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.022932    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.023467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.023480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.023669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.023945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024134    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dwwmk\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024222    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024245    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024321    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lnqjj\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nn9df\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024706    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.024132    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-shm9q\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556tj\" (UniqueName: \"kubernetes.io/projected/5485e8a1-7d02-4313-84d1-d603d7016774-kube-api-access-556tj\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-etc-kubernetes\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-device-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-registration-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025549    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwmls\""
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25378334-29ae-4662-a4ba-9f4b126c8801-konnectivity-ca\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.025865 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-kubelet-config\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-systemd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-cnibin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-kubelet\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-multus-certs\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-iptables-alerter-script\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcjz\" (UniqueName: \"kubernetes.io/projected/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kube-api-access-vvcjz\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-node-log\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jd7rx\""
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-bin\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-socket-dir-parent\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.025925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-netd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026210    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bpnzx\""
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7dt\" (UniqueName: \"kubernetes.io/projected/b805f433-8c13-41a2-8222-5b1caf76fdb2-kube-api-access-5g7dt\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0de05ec0-24fb-4500-a408-3cf08a5b210b-serviceca\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-bin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027162 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-daemon-config\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-socket-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-dbus\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-cnibin\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.026946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-k8s-cni-cncf-io\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-conf-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-config\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-kubelet\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-slash\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-env-overrides\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.027966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-host-slash\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknpf\" (UniqueName: \"kubernetes.io/projected/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-kube-api-access-vknpf\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-netns\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-log-socket\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.027970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-system-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-netns\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-script-lib\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-system-cni-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzbk\" (UniqueName: \"kubernetes.io/projected/4d9be85a-318f-4ab5-aece-2c274cef457d-kube-api-access-wrzbk\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-sys-fs\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25378334-29ae-4662-a4ba-9f4b126c8801-agent-certs\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-hostroot\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-systemd-units\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjgk\" (UniqueName: \"kubernetes.io/projected/0de05ec0-24fb-4500-a408-3cf08a5b210b-kube-api-access-mfjgk\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.029133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-multus\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovn-node-metrics-cert\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-etc-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-ovn\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-cni-binary-copy\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8959\" (UniqueName: \"kubernetes.io/projected/aab7313b-388e-42a1-8f32-3f2e4f594077-kube-api-access-d8959\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-var-lib-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.030053 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.028678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0de05ec0-24fb-4500-a408-3cf08a5b210b-host\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.041901 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.041875    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:13.086367 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.086323    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9z8r"
Sep 09 19:55:13.101068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.101044    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9z8r"
Sep 09 19:55:13.112560 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.112542    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:55:13.123800 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.123761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal" event={"ID":"a5b9edf47ad1b8c3a43ea0ebfaa647f9","Type":"ContainerStarted","Data":"aa83e4cc3fbf8110ad8b1d31d820f03d409e1c4d0d9ff31c7c64bf058b968668"}
Sep 09 19:55:13.128991 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.128963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-etc-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129107 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-ovn\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129107 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-cni-binary-copy\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.129222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-etc-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-ovn\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8959\" (UniqueName: \"kubernetes.io/projected/aab7313b-388e-42a1-8f32-3f2e4f594077-kube-api-access-d8959\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.129222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-systemd\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.129222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-var-lib-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-var-lib-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0de05ec0-24fb-4500-a408-3cf08a5b210b-host\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0de05ec0-24fb-4500-a408-3cf08a5b210b-host\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556tj\" (UniqueName: \"kubernetes.io/projected/5485e8a1-7d02-4313-84d1-d603d7016774-kube-api-access-556tj\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-etc-kubernetes\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-device-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-tmp\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-hosts-file\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.129472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-etc-kubernetes\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-device-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-registration-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25378334-29ae-4662-a4ba-9f4b126c8801-konnectivity-ca\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-registration-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fktv\" (UniqueName: \"kubernetes.io/projected/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-kube-api-access-5fktv\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-kubelet-config\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-systemd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-cnibin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-kubelet-config\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-kubelet\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-systemd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-multus-certs\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-iptables-alerter-script\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-cnibin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-kubelet\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.129992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-multus-certs\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcjz\" (UniqueName: \"kubernetes.io/projected/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kube-api-access-vvcjz\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6sz\" (UniqueName: \"kubernetes.io/projected/11300a66-27d9-4e1e-b83a-3264f33d6a96-kube-api-access-jd6sz\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-node-log\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-node-log\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-bin\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-socket-dir-parent\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-tmp-dir\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-netd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7dt\" (UniqueName: \"kubernetes.io/projected/b805f433-8c13-41a2-8222-5b1caf76fdb2-kube-api-access-5g7dt\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-bin\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-socket-dir-parent\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-cni-netd\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25378334-29ae-4662-a4ba-9f4b126c8801-konnectivity-ca\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.130881 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0de05ec0-24fb-4500-a408-3cf08a5b210b-serviceca\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-bin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-daemon-config\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-socket-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-bin\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-dbus\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-iptables-alerter-script\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-socket-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-cnibin\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-dbus\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-cni-binary-copy\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-k8s-cni-cncf-io\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.130664    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.131743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-conf-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-k8s-cni-cncf-io\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.130729    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.630708849 +0000 UTC m=+3.031386035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-cnibin\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-conf-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-modprobe-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-config\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-kubelet\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-slash\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-kubelet\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0de05ec0-24fb-4500-a408-3cf08a5b210b-serviceca\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-slash\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-run-openvswitch\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.132664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.130976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-env-overrides\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d9be85a-318f-4ab5-aece-2c274cef457d-multus-daemon-config\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-host-slash\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknpf\" (UniqueName: \"kubernetes.io/projected/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-kube-api-access-vknpf\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-run\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-netns\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-log-socket\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-system-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-run-netns\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-netns\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-run-netns\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-log-socket\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.133479 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-conf\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-host\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-host-slash\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-script-lib\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-env-overrides\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-system-cni-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-system-cni-dir\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-config\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5485e8a1-7d02-4313-84d1-d603d7016774-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzbk\" (UniqueName: \"kubernetes.io/projected/4d9be85a-318f-4ab5-aece-2c274cef457d-kube-api-access-wrzbk\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5485e8a1-7d02-4313-84d1-d603d7016774-system-cni-dir\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-sys-fs\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25378334-29ae-4662-a4ba-9f4b126c8801-agent-certs\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-sys-fs\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-tuned\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.134353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-hostroot\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-hostroot\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-kubernetes\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-lib-modules\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-systemd-units\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjgk\" (UniqueName: \"kubernetes.io/projected/0de05ec0-24fb-4500-a408-3cf08a5b210b-kube-api-access-mfjgk\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-multus\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b805f433-8c13-41a2-8222-5b1caf76fdb2-systemd-units\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysconfig\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovnkube-script-lib\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.131952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-sys\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d9be85a-318f-4ab5-aece-2c274cef457d-host-var-lib-cni-multus\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovn-node-metrics-cert\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.132162    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.132220    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.632208002 +0000 UTC m=+3.032885174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.135208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-var-lib-kubelet\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.135978 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.132393    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:55:13.136307 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.136289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b805f433-8c13-41a2-8222-5b1caf76fdb2-ovn-node-metrics-cert\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.136488 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.136472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25378334-29ae-4662-a4ba-9f4b126c8801-agent-certs\") pod \"konnectivity-agent-csjvb\" (UID: \"25378334-29ae-4662-a4ba-9f4b126c8801\") " pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.149249 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.149223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8959\" (UniqueName: \"kubernetes.io/projected/aab7313b-388e-42a1-8f32-3f2e4f594077-kube-api-access-d8959\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.149707 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.149681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556tj\" (UniqueName: \"kubernetes.io/projected/5485e8a1-7d02-4313-84d1-d603d7016774-kube-api-access-556tj\") pod \"multus-additional-cni-plugins-cpkgx\" (UID: \"5485e8a1-7d02-4313-84d1-d603d7016774\") " pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.149993    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.150017    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.150032    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.150113    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.650094774 +0000 UTC m=+3.050771949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.150239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7dt\" (UniqueName: \"kubernetes.io/projected/b805f433-8c13-41a2-8222-5b1caf76fdb2-kube-api-access-5g7dt\") pod \"ovnkube-node-hr562\" (UID: \"b805f433-8c13-41a2-8222-5b1caf76fdb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.150322 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.150266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcjz\" (UniqueName: \"kubernetes.io/projected/3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94-kube-api-access-vvcjz\") pod \"aws-ebs-csi-driver-node-6fww4\" (UID: \"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.152544 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.152513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzbk\" (UniqueName: \"kubernetes.io/projected/4d9be85a-318f-4ab5-aece-2c274cef457d-kube-api-access-wrzbk\") pod \"multus-cb56b\" (UID: \"4d9be85a-318f-4ab5-aece-2c274cef457d\") " pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.155295 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.155271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknpf\" (UniqueName: \"kubernetes.io/projected/5d3a064b-f8e0-481b-9eb0-a25dce78a84b-kube-api-access-vknpf\") pod \"iptables-alerter-smrcb\" (UID: \"5d3a064b-f8e0-481b-9eb0-a25dce78a84b\") " pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.159003 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.158981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjgk\" (UniqueName: \"kubernetes.io/projected/0de05ec0-24fb-4500-a408-3cf08a5b210b-kube-api-access-mfjgk\") pod \"node-ca-tsnld\" (UID: \"0de05ec0-24fb-4500-a408-3cf08a5b210b\") " pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.212855 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.212822    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:13.233117 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-modprobe-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-run\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-conf\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-host\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-run\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-tuned\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-kubernetes\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-lib-modules\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysconfig\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-conf\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-sys\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-host\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-var-lib-kubelet\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-systemd\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-lib-modules\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.233552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-modprobe-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-kubernetes\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-systemd\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-tmp\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-hosts-file\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fktv\" (UniqueName: \"kubernetes.io/projected/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-kube-api-access-5fktv\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-sys\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6sz\" (UniqueName: \"kubernetes.io/projected/11300a66-27d9-4e1e-b83a-3264f33d6a96-kube-api-access-jd6sz\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-var-lib-kubelet\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysconfig\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-tmp-dir\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-hosts-file\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.233825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-sysctl-d\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.234188 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.234122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-tmp-dir\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.236030 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.235963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-etc-tuned\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.236408 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.236322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11300a66-27d9-4e1e-b83a-3264f33d6a96-tmp\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.246258 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.246170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fktv\" (UniqueName: \"kubernetes.io/projected/f145ad01-42a3-4ad7-a6cd-f556d7c99fca-kube-api-access-5fktv\") pod \"node-resolver-cn5pp\" (UID: \"f145ad01-42a3-4ad7-a6cd-f556d7c99fca\") " pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.248251 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.248227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6sz\" (UniqueName: \"kubernetes.io/projected/11300a66-27d9-4e1e-b83a-3264f33d6a96-kube-api-access-jd6sz\") pod \"tuned-zg7cm\" (UID: \"11300a66-27d9-4e1e-b83a-3264f33d6a96\") " pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.313228 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.312964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:13.321171 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.321149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tsnld"
Sep 09 19:55:13.332900 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.332867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpkgx"
Sep 09 19:55:13.333233 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:13.333207    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb805f433_8c13_41a2_8222_5b1caf76fdb2.slice/crio-16bfeaf5b7b4347812065b5895f4a8de03a79727826b4f7df6feb2d429818971 WatchSource:0}: Error finding container 16bfeaf5b7b4347812065b5895f4a8de03a79727826b4f7df6feb2d429818971: Status 404 returned error can't find the container with id 16bfeaf5b7b4347812065b5895f4a8de03a79727826b4f7df6feb2d429818971
Sep 09 19:55:13.334716 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:13.334652    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de05ec0_24fb_4500_a408_3cf08a5b210b.slice/crio-9d8782fcc36ae587a6e199306d48fc092a1961753836302628f1db0fdeef9f6a WatchSource:0}: Error finding container 9d8782fcc36ae587a6e199306d48fc092a1961753836302628f1db0fdeef9f6a: Status 404 returned error can't find the container with id 9d8782fcc36ae587a6e199306d48fc092a1961753836302628f1db0fdeef9f6a
Sep 09 19:55:13.339269 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.339102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cb56b"
Sep 09 19:55:13.341426 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.341195    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:13.348527 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.348365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-smrcb"
Sep 09 19:55:13.355229 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.355208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:13.362688 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.362608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4"
Sep 09 19:55:13.369150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.369128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zg7cm"
Sep 09 19:55:13.375161 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.375140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cn5pp"
Sep 09 19:55:13.636152 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.636036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:13.636152 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.636118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:13.636391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.636209    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.636391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.636235    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.636391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.636290    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:14.636267982 +0000 UTC m=+4.036945183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.636391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.636325    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:14.63630121 +0000 UTC m=+4.036978390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.736508 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:13.736469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:13.736709 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.736669    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:13.736787 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.736719    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:13.736787 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.736735    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:13.736885 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:13.736805    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:14.736789274 +0000 UTC m=+4.137466446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:14.103074 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.102993    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:13 +0000 UTC" deadline="2028-05-19 12:37:21.42204914 +0000 UTC"
Sep 09 19:55:14.103074 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.103023    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14824h42m7.319029382s"
Sep 09 19:55:14.119360 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.119323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:14.119520 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.119426    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:14.127029 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.126980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"16bfeaf5b7b4347812065b5895f4a8de03a79727826b4f7df6feb2d429818971"}
Sep 09 19:55:14.128243 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.128212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tsnld" event={"ID":"0de05ec0-24fb-4500-a408-3cf08a5b210b","Type":"ContainerStarted","Data":"9d8782fcc36ae587a6e199306d48fc092a1961753836302628f1db0fdeef9f6a"}
Sep 09 19:55:14.539998 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:14.539749    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11300a66_27d9_4e1e_b83a_3264f33d6a96.slice/crio-924b9fe9eef4a667dcddee94a351a6ed36257589fff3ff73cf8f1c894a7dc864 WatchSource:0}: Error finding container 924b9fe9eef4a667dcddee94a351a6ed36257589fff3ff73cf8f1c894a7dc864: Status 404 returned error can't find the container with id 924b9fe9eef4a667dcddee94a351a6ed36257589fff3ff73cf8f1c894a7dc864
Sep 09 19:55:14.544114 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:14.544084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf145ad01_42a3_4ad7_a6cd_f556d7c99fca.slice/crio-1e6dc472985bf5c35f7c76cda0e97926322c73d0118a67d139506019206cd009 WatchSource:0}: Error finding container 1e6dc472985bf5c35f7c76cda0e97926322c73d0118a67d139506019206cd009: Status 404 returned error can't find the container with id 1e6dc472985bf5c35f7c76cda0e97926322c73d0118a67d139506019206cd009
Sep 09 19:55:14.546562 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:14.546499    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3a064b_f8e0_481b_9eb0_a25dce78a84b.slice/crio-542bdedb27c47aeb727b76d09274c903283c3e4616aa8a9b43418a7df0374b4c WatchSource:0}: Error finding container 542bdedb27c47aeb727b76d09274c903283c3e4616aa8a9b43418a7df0374b4c: Status 404 returned error can't find the container with id 542bdedb27c47aeb727b76d09274c903283c3e4616aa8a9b43418a7df0374b4c
Sep 09 19:55:14.547641 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:14.547618    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5485e8a1_7d02_4313_84d1_d603d7016774.slice/crio-b9bcd2422f20ec154344b7d761a2dfb2d6beeea4af9554a6a343e96200030f15 WatchSource:0}: Error finding container b9bcd2422f20ec154344b7d761a2dfb2d6beeea4af9554a6a343e96200030f15: Status 404 returned error can't find the container with id b9bcd2422f20ec154344b7d761a2dfb2d6beeea4af9554a6a343e96200030f15
Sep 09 19:55:14.644662 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.644623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:14.644792 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.644742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:14.644848 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.644824    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:14.644993 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.644889    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:14.644993 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.644894    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.644871912 +0000 UTC m=+6.045549097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:14.644993 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.644957    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.644938993 +0000 UTC m=+6.045616168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:14.746143 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:14.746101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:14.746336 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.746305    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:14.746336 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.746339    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:14.746520 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.746353    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:14.746520 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:14.746422    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.746402962 +0000 UTC m=+6.147080159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:15.103990 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.103849    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:13 +0000 UTC" deadline="2028-04-02 09:53:09.611256486 +0000 UTC"
Sep 09 19:55:15.103990 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.103910    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13693h57m54.507350792s"
Sep 09 19:55:15.123035 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.122495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:15.123035 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:15.122638    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:15.123501 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.123314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:15.123501 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:15.123444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:15.136029 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.135996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn5pp" event={"ID":"f145ad01-42a3-4ad7-a6cd-f556d7c99fca","Type":"ContainerStarted","Data":"1e6dc472985bf5c35f7c76cda0e97926322c73d0118a67d139506019206cd009"}
Sep 09 19:55:15.137814 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.137784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg7cm" event={"ID":"11300a66-27d9-4e1e-b83a-3264f33d6a96","Type":"ContainerStarted","Data":"924b9fe9eef4a667dcddee94a351a6ed36257589fff3ff73cf8f1c894a7dc864"}
Sep 09 19:55:15.140986 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.140958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal" event={"ID":"42b537ed4469da6dbc5578dc2b5d442c","Type":"ContainerStarted","Data":"46a1bbf8846b12dcb847b08e3a7bcdade395123e9df17f6374c7a2601c14a692"}
Sep 09 19:55:15.143305 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.143281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-smrcb" event={"ID":"5d3a064b-f8e0-481b-9eb0-a25dce78a84b","Type":"ContainerStarted","Data":"542bdedb27c47aeb727b76d09274c903283c3e4616aa8a9b43418a7df0374b4c"}
Sep 09 19:55:15.155539 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.155513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb56b" event={"ID":"4d9be85a-318f-4ab5-aece-2c274cef457d","Type":"ContainerStarted","Data":"c4bcb77cdd9473938023ef58431f756c31834001c184b85501908434827a5171"}
Sep 09 19:55:15.163727 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.163099    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5b9edf47ad1b8c3a43ea0ebfaa647f9" containerID="add7450f140ba16cdf04b0336900371e1879dd105cfbb1c86fb5ba427e536ed6" exitCode=0
Sep 09 19:55:15.163727 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.163183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal" event={"ID":"a5b9edf47ad1b8c3a43ea0ebfaa647f9","Type":"ContainerDied","Data":"add7450f140ba16cdf04b0336900371e1879dd105cfbb1c86fb5ba427e536ed6"}
Sep 09 19:55:15.166927 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.166647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csjvb" event={"ID":"25378334-29ae-4662-a4ba-9f4b126c8801","Type":"ContainerStarted","Data":"57734d442b662ba48151a364f52364119e85d1b52c13649ed5576785e0f502ea"}
Sep 09 19:55:15.171931 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.171844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4" event={"ID":"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94","Type":"ContainerStarted","Data":"2d72593b83e21eeaa99c0016ed6801f6c1dcceea6eca53639c55b279d10f3e22"}
Sep 09 19:55:15.173836 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.173767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerStarted","Data":"b9bcd2422f20ec154344b7d761a2dfb2d6beeea4af9554a6a343e96200030f15"}
Sep 09 19:55:15.221391 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:15.221327    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-16.ec2.internal" podStartSLOduration=2.221299438 podStartE2EDuration="2.221299438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:15.184229165 +0000 UTC m=+4.584906362" watchObservedRunningTime="2026-09-09 19:55:15.221299438 +0000 UTC m=+4.621976635"
Sep 09 19:55:16.119303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:16.119267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:16.119960 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.119416    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:16.189405 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:16.188926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal" event={"ID":"a5b9edf47ad1b8c3a43ea0ebfaa647f9","Type":"ContainerStarted","Data":"5f9819529c486d0852b9ea14c11cca8142afd49298b16629ca2a6a2ee1bbd769"}
Sep 09 19:55:16.661916 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:16.661875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:16.662097 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:16.662048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:16.662202 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.662184    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:16.662276 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.662251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:20.662230735 +0000 UTC m=+10.062907909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:16.662720 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.662691    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:16.662814 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.662775    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:20.662741681 +0000 UTC m=+10.063418859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:16.763258 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:16.763220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:16.763468 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.763445    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:16.763529 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.763474    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:16.763529 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.763487    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:16.763642 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:16.763549    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:20.763531199 +0000 UTC m=+10.164208376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:17.120366 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:17.120219    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:17.120366 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:17.120258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:17.120915 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:17.120362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:17.120915 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:17.120751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:18.119294 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:18.119259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:18.119492 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:18.119398    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:19.118956 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:19.118917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:19.119432 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:19.118926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:19.119432 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:19.119080    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:19.119432 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:19.119167    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:20.119570 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:20.119479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:20.120093 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.119622    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:20.698074 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:20.698034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:20.698260 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:20.698198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:20.698260 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.698218    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:20.698391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.698296    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:28.698272473 +0000 UTC m=+18.098949653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:20.698391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.698322    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:20.698391 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.698376    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:28.698358126 +0000 UTC m=+18.099035301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:20.799262 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:20.799225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:20.799446 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.799423    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:20.799446 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.799446    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:20.799559 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.799459    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:20.799559 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:20.799516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:28.799496482 +0000 UTC m=+18.200173661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:21.121065 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:21.120508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:21.121065 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:21.120607    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:21.121597 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:21.121153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:21.121597 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:21.121267    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:22.118886 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:22.118852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:22.119039 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:22.118967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:23.118628 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:23.118591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:23.119105 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:23.118769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:23.119105 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:23.118774    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:23.119105 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:23.118889    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:24.118726 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:24.118671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:24.119184 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:24.118834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:25.118983 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:25.118942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:25.119426 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:25.118942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:25.119426 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:25.119084    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:25.119426 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:25.119145    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:26.118755 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:26.118536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:26.118939 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:26.118865    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:27.122585 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:27.122547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:27.123010 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:27.122691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:27.123010 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:27.122705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:27.123010 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:27.122806    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:28.118453 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:28.118419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:28.118632 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.118525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:28.756813 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:28.756778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:28.757284 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.756924    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:28.757284 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:28.756968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:28.757284 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.756992    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:44.756972179 +0000 UTC m=+34.157649365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:28.757284 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.757058    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:28.757284 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.757116    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:44.757097847 +0000 UTC m=+34.157775017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:28.858250 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:28.858215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:28.858430 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.858361    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:28.858430 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.858387    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:28.858430 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.858402    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:28.858586 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:28.858463    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:55:44.858445487 +0000 UTC m=+34.259122670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:29.119237 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:29.119161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:29.119387 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:29.119161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:29.119387 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:29.119300    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:29.119488 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:29.119396    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:30.118694 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:30.118655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:30.119159 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:30.118798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:31.119691 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:31.119634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:31.120132 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:31.119770    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:31.120132 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:31.119795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:31.120132 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:31.119943    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:32.119339 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:32.119302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:32.119482 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:32.119435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:33.119523 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.119314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:33.120411 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.119358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:33.120411 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:33.119634    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:33.120411 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:33.119751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:33.239962 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.239882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tsnld" event={"ID":"0de05ec0-24fb-4500-a408-3cf08a5b210b","Type":"ContainerStarted","Data":"82c154c5e001df82621549da3c70ea10bff35e79085015f600d42bccb2e47fef"}
Sep 09 19:55:33.242933 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.242901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"cdca77a92c2b211f274ae2a75115cf14518f823bd894d25a5b2625668d7194f0"}
Sep 09 19:55:33.243057 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.242965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"94ef9c6e50b9c8b57b71a9b46c8f33ed7e21487c478d910f568bdc6b10ce6b10"}
Sep 09 19:55:33.243057 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.243002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"386eb565dce8d292b2fffa4504032e74570156de2f706d253fcce228cece32d1"}
Sep 09 19:55:33.243057 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.243043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"de15446a20c98c85ede9dd53baedb3264c80257bcfede168737a32b7e8e24138"}
Sep 09 19:55:33.243213 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.243078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"c28ea5e2b7f0b898d2831dd5c7bfe63f64997084fdb0b03ccd85fd7067cc1574"}
Sep 09 19:55:33.243213 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.243123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"51ebfcd158c480c20a96db40ecc8a961b64778748683a4c602d4e235a4e2ea34"}
Sep 09 19:55:33.244512 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.244475    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-csjvb" event={"ID":"25378334-29ae-4662-a4ba-9f4b126c8801","Type":"ContainerStarted","Data":"7faaf88f5a3af8baec20c5a94033094300006bb212b635e958b8811d5812cb1a"}
Sep 09 19:55:33.245812 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.245792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4" event={"ID":"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94","Type":"ContainerStarted","Data":"9f1995923bc6f0336622d0addf7cc5e1b0ac671bf241b8ffde7a42b03fd360ba"}
Sep 09 19:55:33.247063 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.247040    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="5aa1a3fe1f8f680ab49884a2c5c010313a4f1c61ccc7c4f9215e2713df22cc38" exitCode=0
Sep 09 19:55:33.247145 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.247116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"5aa1a3fe1f8f680ab49884a2c5c010313a4f1c61ccc7c4f9215e2713df22cc38"}
Sep 09 19:55:33.248497 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.248460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cn5pp" event={"ID":"f145ad01-42a3-4ad7-a6cd-f556d7c99fca","Type":"ContainerStarted","Data":"2626cbc4f54648a0590de82d546d7af26296d0b7a8e8e7bf7db4e0c4b77c9e2e"}
Sep 09 19:55:33.250009 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.249978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zg7cm" event={"ID":"11300a66-27d9-4e1e-b83a-3264f33d6a96","Type":"ContainerStarted","Data":"3395ab312033d353128a52e4cd319006ee5e0d12cfc467cc4dea44e5410fc796"}
Sep 09 19:55:33.251370 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.251353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cb56b" event={"ID":"4d9be85a-318f-4ab5-aece-2c274cef457d","Type":"ContainerStarted","Data":"2247f4b5ac0db956fb13d39e66f12e70ee7ab39ffe45dc6861e4bab4938abb4c"}
Sep 09 19:55:33.255551 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.255519    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tsnld" podStartSLOduration=6.701194862 podStartE2EDuration="21.255508922s" totalImagesPullingTime="14.55431406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.255159691 +0000 UTC m=+22.655836885" watchObservedRunningTime="2026-09-09 19:55:33.255508922 +0000 UTC m=+22.656186115"
Sep 09 19:55:33.255636 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.255577    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-16.ec2.internal" podStartSLOduration=20.255573601 podStartE2EDuration="20.255573601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:16.209927697 +0000 UTC m=+5.610604894" watchObservedRunningTime="2026-09-09 19:55:33.255573601 +0000 UTC m=+22.656250793"
Sep 09 19:55:33.288051 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.288017    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cn5pp" podStartSLOduration=3.51510782 podStartE2EDuration="21.288005939s" totalImagesPullingTime="17.772898119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.287310438 +0000 UTC m=+22.687987630" watchObservedRunningTime="2026-09-09 19:55:33.288005939 +0000 UTC m=+22.688683131"
Sep 09 19:55:33.306154 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.306104    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cb56b" podStartSLOduration=3.483071348 podStartE2EDuration="21.306088383s" totalImagesPullingTime="17.823017035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.306037043 +0000 UTC m=+22.706714236" watchObservedRunningTime="2026-09-09 19:55:33.306088383 +0000 UTC m=+22.706765579"
Sep 09 19:55:33.319788 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.319748    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-csjvb" podStartSLOduration=3.556499307 podStartE2EDuration="21.319735903s" totalImagesPullingTime="17.763236596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.319734397 +0000 UTC m=+22.720411616" watchObservedRunningTime="2026-09-09 19:55:33.319735903 +0000 UTC m=+22.720413097"
Sep 09 19:55:33.337283 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.337240    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zg7cm" podStartSLOduration=3.547640607 podStartE2EDuration="21.337225092s" totalImagesPullingTime="17.789584485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:33.337183669 +0000 UTC m=+22.737860861" watchObservedRunningTime="2026-09-09 19:55:33.337225092 +0000 UTC m=+22.737902286"
Sep 09 19:55:33.776438 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:33.776236    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:34.087883 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.087726    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:33.776437535Z","UUID":"d722718c-4c78-434e-9bc0-8c86e139944b","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:34.089263 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.089245    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:34.089357 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.089273    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:34.118455 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.118431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:34.118563 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:34.118511    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:34.255528 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.255494    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4" event={"ID":"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94","Type":"ContainerStarted","Data":"95371d2bcb450c2aab1d7845a69dd6aebe832332f0984cc1efb729920f160224"}
Sep 09 19:55:34.256954 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.256933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-smrcb" event={"ID":"5d3a064b-f8e0-481b-9eb0-a25dce78a84b","Type":"ContainerStarted","Data":"e795d0ca7c8d266e6a36136cdb6f1bda21d215d4f21e0701c46e36a7dcfba251"}
Sep 09 19:55:34.278549 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.278514    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-smrcb" podStartSLOduration=4.503579305 podStartE2EDuration="22.27850162s" totalImagesPullingTime="17.774922315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:34.278113265 +0000 UTC m=+23.678790456" watchObservedRunningTime="2026-09-09 19:55:34.27850162 +0000 UTC m=+23.679178812"
Sep 09 19:55:34.909458 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.909428    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:34.910016 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:34.909998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:35.119029 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.119000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:35.119149 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.119031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:35.119202 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:35.119150    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:35.119263 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:35.119244    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:35.264308 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.264212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"7e26a6d4114978a4562728a2385a4e601905d9994e18f6d66f0bfe6ef5b1e38b"}
Sep 09 19:55:35.266587 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.266562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4" event={"ID":"3ff2e3bd-aed0-4ed1-88ca-a9b8501a5e94","Type":"ContainerStarted","Data":"17a0c4765b3144356595da59303f1959c82b5735f1c9f703ca870fd63f97148b"}
Sep 09 19:55:35.267084 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.267055    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:35.267269 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.267253    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-csjvb"
Sep 09 19:55:35.288304 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:35.288262    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6fww4" podStartSLOduration=3.177574494 podStartE2EDuration="23.288248791s" totalImagesPullingTime="20.110674297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:35.287374158 +0000 UTC m=+24.688051350" watchObservedRunningTime="2026-09-09 19:55:35.288248791 +0000 UTC m=+24.688925982"
Sep 09 19:55:36.119068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:36.119034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:36.119239 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:36.119157    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:37.119056 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:37.118810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:37.119056 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:37.118947    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:37.119056 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:37.119040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:37.119559 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:37.119180    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:37.276730 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:37.276553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" event={"ID":"b805f433-8c13-41a2-8222-5b1caf76fdb2","Type":"ContainerStarted","Data":"dc7e98b6917fe5bed6b1e1cc4abf486a9567cfb88592e386b6f0c40c65440dd5"}
Sep 09 19:55:37.303736 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:37.303693    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hr562" podStartSLOduration=6.260553433 podStartE2EDuration="25.303663446s" totalImagesPullingTime="19.043110013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.303640158 +0000 UTC m=+26.704317350" watchObservedRunningTime="2026-09-09 19:55:37.303663446 +0000 UTC m=+26.704340637"
Sep 09 19:55:38.119309 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.119275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:38.119696 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:38.119384    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:38.279107 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.279064    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:38.279277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.279174    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:38.279277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.279226    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:38.295920 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.295891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:38.296205 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:38.296187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:55:39.067412 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.067377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wwrln"]
Sep 09 19:55:39.067528 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.067452    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqdk9"]
Sep 09 19:55:39.067528 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.067523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:39.067667 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:39.067623    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:39.067791 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.067730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:39.067857 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:39.067814    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:39.080570 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.080487    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4zfq"]
Sep 09 19:55:39.080728 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.080591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:39.080728 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:39.080696    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:39.282259 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:39.282231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerStarted","Data":"63210af370aeecda15ca44dedd8f2c9dce7a6d60c1fb144f1cb793ae938c39dc"}
Sep 09 19:55:40.118858 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:40.118821    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:40.119026 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:40.118936    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:40.286447 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:40.286409    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="63210af370aeecda15ca44dedd8f2c9dce7a6d60c1fb144f1cb793ae938c39dc" exitCode=0
Sep 09 19:55:40.286844 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:40.286501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"63210af370aeecda15ca44dedd8f2c9dce7a6d60c1fb144f1cb793ae938c39dc"}
Sep 09 19:55:41.119868 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:41.119701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:41.120028 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:41.119783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:41.120028 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:41.119948    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:41.120108 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:41.120062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:42.118438 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:42.118407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:42.118777 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:42.118500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:43.118648 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:43.118618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:43.119034 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:43.118751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:43.119034 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:43.118797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:43.119034 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:43.118879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:44.118737 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.118635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:44.119148 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.118748    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqdk9" podUID="6eccae26-6ddd-4a56-a10a-f0651e03f40b"
Sep 09 19:55:44.296353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.296322    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="21f1f9f7587c1dc0af512c8495d221164c470e236da3e96e9a14a4f53d17c09b" exitCode=0
Sep 09 19:55:44.296500 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.296368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"21f1f9f7587c1dc0af512c8495d221164c470e236da3e96e9a14a4f53d17c09b"}
Sep 09 19:55:44.777850 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.777805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:44.777850 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.777860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:44.778097 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.777937    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:44.778097 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.777959    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:44.778097 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.777978    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs podName:aab7313b-388e-42a1-8f32-3f2e4f594077 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.777965989 +0000 UTC m=+66.178643159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs") pod "network-metrics-daemon-l4zfq" (UID: "aab7313b-388e-42a1-8f32-3f2e4f594077") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:44.778097 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.778048    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret podName:7c67ceb3-6e5e-4adf-8f2a-a443e346b239 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.778029211 +0000 UTC m=+66.178706382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret") pod "global-pull-secret-syncer-wwrln" (UID: "7c67ceb3-6e5e-4adf-8f2a-a443e346b239") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:44.879305 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:44.879272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:44.879470 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.879446    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:44.879513 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.879478    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:44.879513 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.879492    2582 projected.go:196] Error preparing data for projected volume kube-api-access-5x2k7 for pod openshift-network-diagnostics/network-check-target-xqdk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:44.879574 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:44.879561    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7 podName:6eccae26-6ddd-4a56-a10a-f0651e03f40b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.879537968 +0000 UTC m=+66.280215144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5x2k7" (UniqueName: "kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7") pod "network-check-target-xqdk9" (UID: "6eccae26-6ddd-4a56-a10a-f0651e03f40b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:45.118423 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.118393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:45.118534 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:45.118515    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4zfq" podUID="aab7313b-388e-42a1-8f32-3f2e4f594077"
Sep 09 19:55:45.118575 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.118562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:45.118664 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:45.118648    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wwrln" podUID="7c67ceb3-6e5e-4adf-8f2a-a443e346b239"
Sep 09 19:55:45.300976 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.300892    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="cad0d443349f7b7d648db0ceac62ce4155f1104bc372c1a746823c274879139f" exitCode=0
Sep 09 19:55:45.300976 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.300935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"cad0d443349f7b7d648db0ceac62ce4155f1104bc372c1a746823c274879139f"}
Sep 09 19:55:45.887269 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.887242    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-16.ec2.internal" event="NodeReady"
Sep 09 19:55:45.887377 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.887358    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:45.938083 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.938047    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rq2pt"]
Sep 09 19:55:45.939951 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.939934    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zw5cx"]
Sep 09 19:55:45.940075 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.940061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:45.942110 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.942091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:45.942443 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.942419    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:45.942443 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.942437    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:45.942597 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.942529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:45.942665 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.942652    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdq5v\""
Sep 09 19:55:45.944176 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.944160    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:45.944292 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.944275    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:45.944361 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.944324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shzs9\""
Sep 09 19:55:45.948929 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.948902    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq2pt"]
Sep 09 19:55:45.952853 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:45.952834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw5cx"]
Sep 09 19:55:46.086980 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.086904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9311d573-337d-4a5f-9693-f675a92c0269-cert\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.086980 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.086933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.086980 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.086966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb9d2f-d6e8-402e-9e48-60589350d940-config-volume\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.087200 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.086988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cb9d2f-d6e8-402e-9e48-60589350d940-tmp-dir\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.087200 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.087046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqz9\" (UniqueName: \"kubernetes.io/projected/d4cb9d2f-d6e8-402e-9e48-60589350d940-kube-api-access-8kqz9\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.087200 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.087122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f84\" (UniqueName: \"kubernetes.io/projected/9311d573-337d-4a5f-9693-f675a92c0269-kube-api-access-z9f84\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.119240 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.119214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:55:46.122076 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.122057    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:46.122185 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.122156    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfksv\""
Sep 09 19:55:46.122280 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.122265    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:46.187951 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.187926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9311d573-337d-4a5f-9693-f675a92c0269-cert\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.188058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.187957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.188058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.187980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb9d2f-d6e8-402e-9e48-60589350d940-config-volume\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.188058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.187999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cb9d2f-d6e8-402e-9e48-60589350d940-tmp-dir\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.188058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.188034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqz9\" (UniqueName: \"kubernetes.io/projected/d4cb9d2f-d6e8-402e-9e48-60589350d940-kube-api-access-8kqz9\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.188225 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.188066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f84\" (UniqueName: \"kubernetes.io/projected/9311d573-337d-4a5f-9693-f675a92c0269-kube-api-access-z9f84\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.188225 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:46.188089    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.188225 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:46.188149    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.688131783 +0000 UTC m=+36.088808970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.188353 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.188329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cb9d2f-d6e8-402e-9e48-60589350d940-tmp-dir\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.188588 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.188570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb9d2f-d6e8-402e-9e48-60589350d940-config-volume\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.192534 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.192411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9311d573-337d-4a5f-9693-f675a92c0269-cert\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.196579 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.196552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqz9\" (UniqueName: \"kubernetes.io/projected/d4cb9d2f-d6e8-402e-9e48-60589350d940-kube-api-access-8kqz9\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.196579 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.196570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f84\" (UniqueName: \"kubernetes.io/projected/9311d573-337d-4a5f-9693-f675a92c0269-kube-api-access-z9f84\") pod \"ingress-canary-rq2pt\" (UID: \"9311d573-337d-4a5f-9693-f675a92c0269\") " pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.250898 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.250864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq2pt"
Sep 09 19:55:46.527060 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.527023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq2pt"]
Sep 09 19:55:46.532775 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:46.532748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9311d573_337d_4a5f_9693_f675a92c0269.slice/crio-30a1857724dfa1a49d19b0e62ecc6fc29468052c2ab64682e5d77dc1725eca29 WatchSource:0}: Error finding container 30a1857724dfa1a49d19b0e62ecc6fc29468052c2ab64682e5d77dc1725eca29: Status 404 returned error can't find the container with id 30a1857724dfa1a49d19b0e62ecc6fc29468052c2ab64682e5d77dc1725eca29
Sep 09 19:55:46.690043 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:46.690012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:46.690185 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:46.690164    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:46.690277 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:46.690266    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.690249021 +0000 UTC m=+37.090926191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:47.119089 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.119054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:55:47.119272 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.119057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:55:47.122247 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.122221    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:47.123057 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.122985    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r2s2\""
Sep 09 19:55:47.123174 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.123075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:47.308749 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.308714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq2pt" event={"ID":"9311d573-337d-4a5f-9693-f675a92c0269","Type":"ContainerStarted","Data":"30a1857724dfa1a49d19b0e62ecc6fc29468052c2ab64682e5d77dc1725eca29"}
Sep 09 19:55:47.697570 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:47.697540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:47.698037 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:47.697728    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:47.698037 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:47.697804    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:49.697783779 +0000 UTC m=+39.098460949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.315718 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.315665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq2pt" event={"ID":"9311d573-337d-4a5f-9693-f675a92c0269","Type":"ContainerStarted","Data":"8d8fd8d65982c1112c0f5f9b2d5ecfb3919f192714845aab5aa0276f692ea5e7"}
Sep 09 19:55:49.333488 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.333417    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rq2pt" podStartSLOduration=2.097779671 podStartE2EDuration="4.333396713s" totalImagesPullingTime="2.235617042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:49.332506134 +0000 UTC m=+38.733183340" watchObservedRunningTime="2026-09-09 19:55:49.333396713 +0000 UTC m=+38.734073907"
Sep 09 19:55:49.621908 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.621873    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"]
Sep 09 19:55:49.643040 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.643011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"]
Sep 09 19:55:49.643204 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.643144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.645868 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.645616    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:49.645868 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.645707    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r6949\""
Sep 09 19:55:49.645868 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.645719    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:49.646088 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.645945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 19:55:49.646088 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.645951    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 19:55:49.713072 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.713044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:49.713267 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:49.713176    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.713267 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:49.713234    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.713217433 +0000 UTC m=+43.113894602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.814017 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.813977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57lx\" (UniqueName: \"kubernetes.io/projected/9b080784-3270-422d-bced-001ff0632258-kube-api-access-r57lx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.814208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.814076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b080784-3270-422d-bced-001ff0632258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.814208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.814098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b080784-3270-422d-bced-001ff0632258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.827397 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.827370    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:55:49.846783 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.846753    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:55:49.846948 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.846863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:49.849222 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.849197    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:55:49.849585 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.849561    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:55:49.849839 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.849825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j6mws\""
Sep 09 19:55:49.850055 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.850039    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:55:49.855533 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.855512    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:55:49.915419 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.915354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57lx\" (UniqueName: \"kubernetes.io/projected/9b080784-3270-422d-bced-001ff0632258-kube-api-access-r57lx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.915419 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.915406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b080784-3270-422d-bced-001ff0632258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.915613 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.915432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b080784-3270-422d-bced-001ff0632258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.916125 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.916094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b080784-3270-422d-bced-001ff0632258-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.918655 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.918636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b080784-3270-422d-bced-001ff0632258-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.923448 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.923423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57lx\" (UniqueName: \"kubernetes.io/projected/9b080784-3270-422d-bced-001ff0632258-kube-api-access-r57lx\") pod \"kube-storage-version-migrator-operator-8d7dd4675-r7ll8\" (UID: \"9b080784-3270-422d-bced-001ff0632258\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:49.953453 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:49.953426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"
Sep 09 19:55:50.016335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016513 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmct2\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016513 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016513 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.016513 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.016454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.117747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.117747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmct2\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.117947    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.117967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118008 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.117976    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfbc545f6-jjpvw: secret "image-registry-tls" not found
Sep 09 19:55:50.118375 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.118047    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls podName:6b226cb1-db1d-4928-98bd-2cc6ebcd4c90 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.618021705 +0000 UTC m=+40.018698974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls") pod "image-registry-6dfbc545f6-jjpvw" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90") : secret "image-registry-tls" not found
Sep 09 19:55:50.118375 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.118277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.118824 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.118795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.119197 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.119177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.120854 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.120831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.120961 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.120837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.127109 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.127076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.127306 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.127289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmct2\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.242262 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.242227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8"]
Sep 09 19:55:50.622102 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:50.622020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:50.622536 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.622167    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:50.622536 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.622186    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfbc545f6-jjpvw: secret "image-registry-tls" not found
Sep 09 19:55:50.622536 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:50.622251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls podName:6b226cb1-db1d-4928-98bd-2cc6ebcd4c90 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:51.622236881 +0000 UTC m=+41.022914056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls") pod "image-registry-6dfbc545f6-jjpvw" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90") : secret "image-registry-tls" not found
Sep 09 19:55:51.598077 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:51.598040    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b080784_3270_422d_bced_001ff0632258.slice/crio-ff1ada1d77ce86b70269cdc9f893feeccd5841bf9c32d9b3150cf761ea2732ff WatchSource:0}: Error finding container ff1ada1d77ce86b70269cdc9f893feeccd5841bf9c32d9b3150cf761ea2732ff: Status 404 returned error can't find the container with id ff1ada1d77ce86b70269cdc9f893feeccd5841bf9c32d9b3150cf761ea2732ff
Sep 09 19:55:51.630902 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:51.630869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:51.631227 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:51.631013    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:51.631227 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:51.631030    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfbc545f6-jjpvw: secret "image-registry-tls" not found
Sep 09 19:55:51.631227 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:51.631078    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls podName:6b226cb1-db1d-4928-98bd-2cc6ebcd4c90 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.631064458 +0000 UTC m=+43.031741627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls") pod "image-registry-6dfbc545f6-jjpvw" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90") : secret "image-registry-tls" not found
Sep 09 19:55:52.327515 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:52.327426    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="1220116c3d65c4d3741349c88250170a1e54febbc0ab5d2374ac270feb372993" exitCode=0
Sep 09 19:55:52.327666 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:52.327529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"1220116c3d65c4d3741349c88250170a1e54febbc0ab5d2374ac270feb372993"}
Sep 09 19:55:52.329168 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:52.329140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8" event={"ID":"9b080784-3270-422d-bced-001ff0632258","Type":"ContainerStarted","Data":"ff1ada1d77ce86b70269cdc9f893feeccd5841bf9c32d9b3150cf761ea2732ff"}
Sep 09 19:55:53.336070 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:53.336031    2582 generic.go:356] "Generic (PLEG): container finished" podID="5485e8a1-7d02-4313-84d1-d603d7016774" containerID="d83b9e4ca0dd6dd503ed0d5714d4d234867aad563e2c9442479399524cdce48d" exitCode=0
Sep 09 19:55:53.336496 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:53.336088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerDied","Data":"d83b9e4ca0dd6dd503ed0d5714d4d234867aad563e2c9442479399524cdce48d"}
Sep 09 19:55:53.647610 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:53.647582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:53.647780 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:53.647762    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:53.647848 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:53.647783    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfbc545f6-jjpvw: secret "image-registry-tls" not found
Sep 09 19:55:53.647848 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:53.647842    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls podName:6b226cb1-db1d-4928-98bd-2cc6ebcd4c90 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:57.647822537 +0000 UTC m=+47.048499721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls") pod "image-registry-6dfbc545f6-jjpvw" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90") : secret "image-registry-tls" not found
Sep 09 19:55:53.748599 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:53.748569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:55:53.748747 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:53.748719    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:53.748907 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:53.748769    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.748755469 +0000 UTC m=+51.149432638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:54.339805 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:54.339767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8" event={"ID":"9b080784-3270-422d-bced-001ff0632258","Type":"ContainerStarted","Data":"55f9983d52941c71485aa55f76aab4648ee6cc6829278825b88c379c89e12fe2"}
Sep 09 19:55:54.343181 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:54.343153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" event={"ID":"5485e8a1-7d02-4313-84d1-d603d7016774","Type":"ContainerStarted","Data":"e46ff84201d15792657364167d61c769ce3d69aa28231f8b56ffa10f9f755c76"}
Sep 09 19:55:54.356077 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:54.356021    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8" podStartSLOduration=3.104963475 podStartE2EDuration="5.356006142s" totalImagesPullingTime="2.251042667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:54.355340442 +0000 UTC m=+43.756017633" watchObservedRunningTime="2026-09-09 19:55:54.356006142 +0000 UTC m=+43.756683334"
Sep 09 19:55:54.375352 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:54.375293    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cpkgx" podStartSLOduration=8.46286213 podStartE2EDuration="42.375272533s" totalImagesPullingTime="33.912410403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:54.373264027 +0000 UTC m=+43.773941212" watchObservedRunningTime="2026-09-09 19:55:54.375272533 +0000 UTC m=+43.775949726"
Sep 09 19:55:55.268412 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.268376    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vkddx"]
Sep 09 19:55:55.288875 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.288830    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkddx"]
Sep 09 19:55:55.289041 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.288943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.291812 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.291789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57xxp\""
Sep 09 19:55:55.291942 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.291791    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:55:55.291942 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.291792    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:55:55.292611 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.292591    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:55:55.292611 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.292602    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:55:55.462067 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.462026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.462482 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.462076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.462482 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.462195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b99cff65-7625-4001-835b-52fe979042d4-crio-socket\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.462482 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.462248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b99cff65-7625-4001-835b-52fe979042d4-data-volume\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.462482 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.462278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxz7\" (UniqueName: \"kubernetes.io/projected/b99cff65-7625-4001-835b-52fe979042d4-kube-api-access-djxz7\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.552320 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.552241    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"]
Sep 09 19:55:55.563358 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563358 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563538 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b99cff65-7625-4001-835b-52fe979042d4-crio-socket\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563538 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:55.563454    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.563538 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b99cff65-7625-4001-835b-52fe979042d4-data-volume\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563538 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxz7\" (UniqueName: \"kubernetes.io/projected/b99cff65-7625-4001-835b-52fe979042d4-kube-api-access-djxz7\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563538 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:55.563530    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.063508862 +0000 UTC m=+45.464186051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.563769 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:55.563457    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.563769 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:55.563630    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.063612766 +0000 UTC m=+45.464289959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.563769 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b99cff65-7625-4001-835b-52fe979042d4-crio-socket\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.563884 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.563856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b99cff65-7625-4001-835b-52fe979042d4-data-volume\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.569575 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.569547    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"]
Sep 09 19:55:55.569700 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.569658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"
Sep 09 19:55:55.572512 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.572495    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:55.572634 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.572600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 19:55:55.572916 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.572898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ng847\""
Sep 09 19:55:55.579194 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.579166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxz7\" (UniqueName: \"kubernetes.io/projected/b99cff65-7625-4001-835b-52fe979042d4-kube-api-access-djxz7\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:55.764484 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.764454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntcj\" (UniqueName: \"kubernetes.io/projected/5f05fec7-154a-4d14-8103-e2561e53a93f-kube-api-access-bntcj\") pod \"migrator-fdbb8fdff-qnh44\" (UID: \"5f05fec7-154a-4d14-8103-e2561e53a93f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"
Sep 09 19:55:55.865343 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.865259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntcj\" (UniqueName: \"kubernetes.io/projected/5f05fec7-154a-4d14-8103-e2561e53a93f-kube-api-access-bntcj\") pod \"migrator-fdbb8fdff-qnh44\" (UID: \"5f05fec7-154a-4d14-8103-e2561e53a93f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"
Sep 09 19:55:55.874305 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.874267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntcj\" (UniqueName: \"kubernetes.io/projected/5f05fec7-154a-4d14-8103-e2561e53a93f-kube-api-access-bntcj\") pod \"migrator-fdbb8fdff-qnh44\" (UID: \"5f05fec7-154a-4d14-8103-e2561e53a93f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"
Sep 09 19:55:55.879006 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:55.878989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"
Sep 09 19:55:56.066766 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:56.066738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:56.066766 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:56.066767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:56.066937 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:56.066859    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:56.066937 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:56.066927    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:57.066911291 +0000 UTC m=+46.467588461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:56.067020 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:56.066866    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:56.067020 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:56.067016    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:57.066998365 +0000 UTC m=+46.467675538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:56.162272 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:56.162194    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44"]
Sep 09 19:55:56.166364 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:55:56.166337    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f05fec7_154a_4d14_8103_e2561e53a93f.slice/crio-fed7001c598887724817a4fe1c649ebb9bc86448fe8534669d4e2fb5aec52317 WatchSource:0}: Error finding container fed7001c598887724817a4fe1c649ebb9bc86448fe8534669d4e2fb5aec52317: Status 404 returned error can't find the container with id fed7001c598887724817a4fe1c649ebb9bc86448fe8534669d4e2fb5aec52317
Sep 09 19:55:56.348509 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:56.348476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44" event={"ID":"5f05fec7-154a-4d14-8103-e2561e53a93f","Type":"ContainerStarted","Data":"fed7001c598887724817a4fe1c649ebb9bc86448fe8534669d4e2fb5aec52317"}
Sep 09 19:55:57.007834 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:57.007803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn5pp_f145ad01-42a3-4ad7-a6cd-f556d7c99fca/dns-node-resolver/0.log"
Sep 09 19:55:57.073623 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:57.073587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:57.073623 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:57.073626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:57.073864 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.073727    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:57.073864 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.073774    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:57.073864 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.073809    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:59.073791836 +0000 UTC m=+48.474469010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:57.073864 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.073841    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:59.073824421 +0000 UTC m=+48.474501591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:57.677193 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:57.677166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:55:57.677325 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.677301    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:57.677325 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.677320    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfbc545f6-jjpvw: secret "image-registry-tls" not found
Sep 09 19:55:57.677439 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:57.677365    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls podName:6b226cb1-db1d-4928-98bd-2cc6ebcd4c90 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.677352273 +0000 UTC m=+55.078029443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls") pod "image-registry-6dfbc545f6-jjpvw" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90") : secret "image-registry-tls" not found
Sep 09 19:55:58.354584 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:58.354543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44" event={"ID":"5f05fec7-154a-4d14-8103-e2561e53a93f","Type":"ContainerStarted","Data":"1d828b63874b00c9af02429b49d81771bb9efa50d693f2857bcf3f39417ea408"}
Sep 09 19:55:58.354982 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:58.354596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44" event={"ID":"5f05fec7-154a-4d14-8103-e2561e53a93f","Type":"ContainerStarted","Data":"0f32a49ba41ec76b19639587a96e966f629e227b1b53071a62477045d3589694"}
Sep 09 19:55:58.371065 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:58.371017    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-qnh44" podStartSLOduration=1.875734165 podStartE2EDuration="3.370998877s" totalImagesPullingTime="1.495264712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:58.370646917 +0000 UTC m=+47.771324110" watchObservedRunningTime="2026-09-09 19:55:58.370998877 +0000 UTC m=+47.771676070"
Sep 09 19:55:58.406950 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:58.406927    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tsnld_0de05ec0-24fb-4500-a408-3cf08a5b210b/node-ca/0.log"
Sep 09 19:55:59.007903 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:59.007870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq2pt_9311d573-337d-4a5f-9693-f675a92c0269/serve-healthcheck-canary/0.log"
Sep 09 19:55:59.088225 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:59.088193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:59.088375 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:55:59.088228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:55:59.088375 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:59.088328    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:59.088484 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:59.088393    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:59.088484 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:59.088405    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.088384976 +0000 UTC m=+52.489062158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:59.088484 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:55:59.088435    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.088424345 +0000 UTC m=+52.489101519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:01.806118 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:01.806074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:01.806612 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:01.806242    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:01.806612 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:01.806309    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls podName:d4cb9d2f-d6e8-402e-9e48-60589350d940 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:17.806291736 +0000 UTC m=+67.206968910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls") pod "dns-default-zw5cx" (UID: "d4cb9d2f-d6e8-402e-9e48-60589350d940") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:03.116408 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:03.116371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:03.116408 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:03.116407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:03.116877 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:03.116494    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:03.116877 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:03.116512    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:03.116877 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:03.116561    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:11.11654314 +0000 UTC m=+60.517220339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:03.116877 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:03.116575    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:11.116569575 +0000 UTC m=+60.517246745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:05.731972 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:05.731935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:05.734352 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:05.734328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"image-registry-6dfbc545f6-jjpvw\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") " pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:05.758302 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:05.758271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:06.026153 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:06.026065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:56:06.028641 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:06.028608    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b226cb1_db1d_4928_98bd_2cc6ebcd4c90.slice/crio-73c6e3916ed809067e2fb6ddf78fa64b9c42b9bdd0eec6c61579aba63d0aa762 WatchSource:0}: Error finding container 73c6e3916ed809067e2fb6ddf78fa64b9c42b9bdd0eec6c61579aba63d0aa762: Status 404 returned error can't find the container with id 73c6e3916ed809067e2fb6ddf78fa64b9c42b9bdd0eec6c61579aba63d0aa762
Sep 09 19:56:06.375340 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:06.375269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" event={"ID":"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90","Type":"ContainerStarted","Data":"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"}
Sep 09 19:56:06.375340 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:06.375318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" event={"ID":"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90","Type":"ContainerStarted","Data":"73c6e3916ed809067e2fb6ddf78fa64b9c42b9bdd0eec6c61579aba63d0aa762"}
Sep 09 19:56:06.375522 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:06.375403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:06.397392 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:06.397346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" podStartSLOduration=17.39733006 podStartE2EDuration="17.39733006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:06.396256987 +0000 UTC m=+55.796934179" watchObservedRunningTime="2026-09-09 19:56:06.39733006 +0000 UTC m=+55.798007251"
Sep 09 19:56:10.297189 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:10.297163    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr562"
Sep 09 19:56:11.174070 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:11.174040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:11.174070 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:11.174078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:11.174306 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:11.174274    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:11.174362 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:11.174346    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls podName:b99cff65-7625-4001-835b-52fe979042d4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:27.174324636 +0000 UTC m=+76.575001828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkddx" (UID: "b99cff65-7625-4001-835b-52fe979042d4") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:11.174522 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:11.174500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b99cff65-7625-4001-835b-52fe979042d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:16.816947 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.816897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:56:16.817514 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.816982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:56:16.819586 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.819567    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:16.819646 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.819576    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:16.829914 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.829883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c67ceb3-6e5e-4adf-8f2a-a443e346b239-original-pull-secret\") pod \"global-pull-secret-syncer-wwrln\" (UID: \"7c67ceb3-6e5e-4adf-8f2a-a443e346b239\") " pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:56:16.830023 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.829937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aab7313b-388e-42a1-8f32-3f2e4f594077-metrics-certs\") pod \"network-metrics-daemon-l4zfq\" (UID: \"aab7313b-388e-42a1-8f32-3f2e4f594077\") " pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:56:16.832850 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.832824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wwrln"
Sep 09 19:56:16.840733 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.840717    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r2s2\""
Sep 09 19:56:16.848671 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.848651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4zfq"
Sep 09 19:56:16.918230 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.918158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:56:16.921324 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.921276    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:56:16.931669 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.931649    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:56:16.941968 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:16.941951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2k7\" (UniqueName: \"kubernetes.io/projected/6eccae26-6ddd-4a56-a10a-f0651e03f40b-kube-api-access-5x2k7\") pod \"network-check-target-xqdk9\" (UID: \"6eccae26-6ddd-4a56-a10a-f0651e03f40b\") " pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:56:17.030427 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.030406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfksv\""
Sep 09 19:56:17.037793 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.037777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:56:17.102993 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.102967    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wwrln"]
Sep 09 19:56:17.106073 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:17.106040    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c67ceb3_6e5e_4adf_8f2a_a443e346b239.slice/crio-da5c433b82ba6d9d6e2ea19b162b9519e0ecaf7a8d38382d6e45e1952b5f03c4 WatchSource:0}: Error finding container da5c433b82ba6d9d6e2ea19b162b9519e0ecaf7a8d38382d6e45e1952b5f03c4: Status 404 returned error can't find the container with id da5c433b82ba6d9d6e2ea19b162b9519e0ecaf7a8d38382d6e45e1952b5f03c4
Sep 09 19:56:17.126664 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.126640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4zfq"]
Sep 09 19:56:17.129413 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:17.129387    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab7313b_388e_42a1_8f32_3f2e4f594077.slice/crio-7f55e5de8c26263918eb8e12099ae8bda0863a8faf3dadcccd78a16c30b60210 WatchSource:0}: Error finding container 7f55e5de8c26263918eb8e12099ae8bda0863a8faf3dadcccd78a16c30b60210: Status 404 returned error can't find the container with id 7f55e5de8c26263918eb8e12099ae8bda0863a8faf3dadcccd78a16c30b60210
Sep 09 19:56:17.304605 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.304575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqdk9"]
Sep 09 19:56:17.307204 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:17.307179    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eccae26_6ddd_4a56_a10a_f0651e03f40b.slice/crio-2da7423d658152346565fa7d1fda731679931b7d61f3a55f14b292d300e231f5 WatchSource:0}: Error finding container 2da7423d658152346565fa7d1fda731679931b7d61f3a55f14b292d300e231f5: Status 404 returned error can't find the container with id 2da7423d658152346565fa7d1fda731679931b7d61f3a55f14b292d300e231f5
Sep 09 19:56:17.401514 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.401437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4zfq" event={"ID":"aab7313b-388e-42a1-8f32-3f2e4f594077","Type":"ContainerStarted","Data":"7f55e5de8c26263918eb8e12099ae8bda0863a8faf3dadcccd78a16c30b60210"}
Sep 09 19:56:17.402622 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.402594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wwrln" event={"ID":"7c67ceb3-6e5e-4adf-8f2a-a443e346b239","Type":"ContainerStarted","Data":"da5c433b82ba6d9d6e2ea19b162b9519e0ecaf7a8d38382d6e45e1952b5f03c4"}
Sep 09 19:56:17.403640 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.403620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqdk9" event={"ID":"6eccae26-6ddd-4a56-a10a-f0651e03f40b","Type":"ContainerStarted","Data":"2da7423d658152346565fa7d1fda731679931b7d61f3a55f14b292d300e231f5"}
Sep 09 19:56:17.824160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.824124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:17.826631 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:17.826608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4cb9d2f-d6e8-402e-9e48-60589350d940-metrics-tls\") pod \"dns-default-zw5cx\" (UID: \"d4cb9d2f-d6e8-402e-9e48-60589350d940\") " pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:18.059826 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.059595    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shzs9\""
Sep 09 19:56:18.067762 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.067608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:18.370959 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.370881    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw5cx"]
Sep 09 19:56:18.375218 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:18.375189    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cb9d2f_d6e8_402e_9e48_60589350d940.slice/crio-b5199a69ce46a73c69bc535759c7168c93611d73655ccd783d2a03953d1ba662 WatchSource:0}: Error finding container b5199a69ce46a73c69bc535759c7168c93611d73655ccd783d2a03953d1ba662: Status 404 returned error can't find the container with id b5199a69ce46a73c69bc535759c7168c93611d73655ccd783d2a03953d1ba662
Sep 09 19:56:18.409013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.408974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw5cx" event={"ID":"d4cb9d2f-d6e8-402e-9e48-60589350d940","Type":"ContainerStarted","Data":"b5199a69ce46a73c69bc535759c7168c93611d73655ccd783d2a03953d1ba662"}
Sep 09 19:56:18.411077 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.411050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4zfq" event={"ID":"aab7313b-388e-42a1-8f32-3f2e4f594077","Type":"ContainerStarted","Data":"a1f04fff2a16070d8a75a6b8aa55459dae91c7f37f77d2afe4443709eeddbe31"}
Sep 09 19:56:18.411208 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.411103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4zfq" event={"ID":"aab7313b-388e-42a1-8f32-3f2e4f594077","Type":"ContainerStarted","Data":"1d9759ea5c6c8fa4fb6c9a957ae4664d4340826cf08fc86785d514ce0ae2d205"}
Sep 09 19:56:18.428227 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:18.428178    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l4zfq" podStartSLOduration=65.440488271 podStartE2EDuration="1m6.428156699s" totalImagesPullingTime="987.668428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:18.427632334 +0000 UTC m=+67.828309525" watchObservedRunningTime="2026-09-09 19:56:18.428156699 +0000 UTC m=+67.828833893"
Sep 09 19:56:21.428282 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:21.428237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqdk9" event={"ID":"6eccae26-6ddd-4a56-a10a-f0651e03f40b","Type":"ContainerStarted","Data":"c7cbb4ed85e5d4a56ab61b6f690345c1c21de7f7150e440e9335216a4a3cb017"}
Sep 09 19:56:21.428851 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:21.428410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:56:21.447358 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:21.447118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xqdk9" podStartSLOduration=66.386822062 podStartE2EDuration="1m9.447088548s" totalImagesPullingTime="3.060266486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:21.445772461 +0000 UTC m=+70.846449651" watchObservedRunningTime="2026-09-09 19:56:21.447088548 +0000 UTC m=+70.847765741"
Sep 09 19:56:22.434160 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:22.434123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw5cx" event={"ID":"d4cb9d2f-d6e8-402e-9e48-60589350d940","Type":"ContainerStarted","Data":"95faa4c775e03436db7ad880da6b787468f936cd0d594af886ab5aea06a56a17"}
Sep 09 19:56:22.434707 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:22.434194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw5cx" event={"ID":"d4cb9d2f-d6e8-402e-9e48-60589350d940","Type":"ContainerStarted","Data":"1f78c339062c20dc2c8588b5980d13a819b2994757aa18adaffa032a1a4ffba9"}
Sep 09 19:56:22.462447 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:22.462403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zw5cx" podStartSLOduration=33.973176378 podStartE2EDuration="37.462387374s" totalImagesPullingTime="3.489210996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.461872284 +0000 UTC m=+71.862549477" watchObservedRunningTime="2026-09-09 19:56:22.462387374 +0000 UTC m=+71.863064607"
Sep 09 19:56:23.439027 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.438911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:23.648556 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.648521    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"]
Sep 09 19:56:23.680817 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.680784    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:56:23.680989 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.680845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"]
Sep 09 19:56:23.680989 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.680931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.684870 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.684738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q28n8\""
Sep 09 19:56:23.685722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.685420    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:23.686461 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.686444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:23.686563 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.686448    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 19:56:23.686750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.686442    2582 patch_prober.go:28] interesting pod/image-registry-6dfbc545f6-jjpvw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 19:56:23.686750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.686662    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 19:56:23.687706 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.687664    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:23.745116 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.745074    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-g68c4"]
Sep 09 19:56:23.764838 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.764809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:23.767627 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.767601    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bm48h\""
Sep 09 19:56:23.768234 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.768213    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 19:56:23.768429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.768402    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 19:56:23.774221 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.774201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e15636-ee50-4dd7-9497-49684e67f1a6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.774300 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.774241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzm26\" (UniqueName: \"kubernetes.io/projected/74e15636-ee50-4dd7-9497-49684e67f1a6-kube-api-access-pzm26\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.781214 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.781190    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-g68c4"]
Sep 09 19:56:23.848554 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.848522    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b87587df-pzhtd"]
Sep 09 19:56:23.863433 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.863407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.866978 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.866953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b87587df-pzhtd"]
Sep 09 19:56:23.875079 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.875054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e15636-ee50-4dd7-9497-49684e67f1a6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.875169 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.875089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngsh\" (UniqueName: \"kubernetes.io/projected/895a680f-2d1a-4c04-b9c9-7be40f8cd95c-kube-api-access-kngsh\") pod \"downloads-85d98c998f-g68c4\" (UID: \"895a680f-2d1a-4c04-b9c9-7be40f8cd95c\") " pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:23.875169 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.875126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzm26\" (UniqueName: \"kubernetes.io/projected/74e15636-ee50-4dd7-9497-49684e67f1a6-kube-api-access-pzm26\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.877438 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.877420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e15636-ee50-4dd7-9497-49684e67f1a6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.914307 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.914285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzm26\" (UniqueName: \"kubernetes.io/projected/74e15636-ee50-4dd7-9497-49684e67f1a6-kube-api-access-pzm26\") pod \"managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq\" (UID: \"74e15636-ee50-4dd7-9497-49684e67f1a6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:23.976492 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-trusted-ca\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976622 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngsh\" (UniqueName: \"kubernetes.io/projected/895a680f-2d1a-4c04-b9c9-7be40f8cd95c-kube-api-access-kngsh\") pod \"downloads-85d98c998f-g68c4\" (UID: \"895a680f-2d1a-4c04-b9c9-7be40f8cd95c\") " pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:23.976622 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-image-registry-private-configuration\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976622 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-tls\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-certificates\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-installation-pull-secrets\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-bound-sa-token\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976839 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81eed42f-a908-4013-bbd4-ceaa766a77f4-ca-trust-extracted\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.976839 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.976782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68mf\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-kube-api-access-b68mf\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:23.989263 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.989242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngsh\" (UniqueName: \"kubernetes.io/projected/895a680f-2d1a-4c04-b9c9-7be40f8cd95c-kube-api-access-kngsh\") pod \"downloads-85d98c998f-g68c4\" (UID: \"895a680f-2d1a-4c04-b9c9-7be40f8cd95c\") " pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:23.998967 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:23.998948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"
Sep 09 19:56:24.075099 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.075072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:24.077649 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-bound-sa-token\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077718 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81eed42f-a908-4013-bbd4-ceaa766a77f4-ca-trust-extracted\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077718 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68mf\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-kube-api-access-b68mf\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077798 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-trusted-ca\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077798 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-image-registry-private-configuration\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077889 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-tls\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077889 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-certificates\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.077889 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.077848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-installation-pull-secrets\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.078107 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.078084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81eed42f-a908-4013-bbd4-ceaa766a77f4-ca-trust-extracted\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.078634 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.078611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-certificates\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.078761 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.078737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81eed42f-a908-4013-bbd4-ceaa766a77f4-trusted-ca\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.080281 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.080260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-image-registry-private-configuration\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.080372 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.080356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81eed42f-a908-4013-bbd4-ceaa766a77f4-installation-pull-secrets\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.080454 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.080433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-registry-tls\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.087541 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.087513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68mf\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-kube-api-access-b68mf\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.087635 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.087619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eed42f-a908-4013-bbd4-ceaa766a77f4-bound-sa-token\") pod \"image-registry-57b87587df-pzhtd\" (UID: \"81eed42f-a908-4013-bbd4-ceaa766a77f4\") " pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.172289 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.172254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:24.407423 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.407394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq"]
Sep 09 19:56:24.409276 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.409256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-g68c4"]
Sep 09 19:56:24.411280 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:24.411258    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74e15636_ee50_4dd7_9497_49684e67f1a6.slice/crio-903c4aac269ee633f693cbcdd50c680d1efd10956dcc735bc4333804d9c43e9d WatchSource:0}: Error finding container 903c4aac269ee633f693cbcdd50c680d1efd10956dcc735bc4333804d9c43e9d: Status 404 returned error can't find the container with id 903c4aac269ee633f693cbcdd50c680d1efd10956dcc735bc4333804d9c43e9d
Sep 09 19:56:24.412805 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:24.412778    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895a680f_2d1a_4c04_b9c9_7be40f8cd95c.slice/crio-34cb25cb5a26916d380225efa5896a11503fd8fed2b39b6d01eadfd126be4134 WatchSource:0}: Error finding container 34cb25cb5a26916d380225efa5896a11503fd8fed2b39b6d01eadfd126be4134: Status 404 returned error can't find the container with id 34cb25cb5a26916d380225efa5896a11503fd8fed2b39b6d01eadfd126be4134
Sep 09 19:56:24.442845 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.442818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wwrln" event={"ID":"7c67ceb3-6e5e-4adf-8f2a-a443e346b239","Type":"ContainerStarted","Data":"6b40500712a44b8a85b2db620ea404c45ceb839f6c7c7cafa80f7f468260d3d5"}
Sep 09 19:56:24.444084 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.444059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq" event={"ID":"74e15636-ee50-4dd7-9497-49684e67f1a6","Type":"ContainerStarted","Data":"903c4aac269ee633f693cbcdd50c680d1efd10956dcc735bc4333804d9c43e9d"}
Sep 09 19:56:24.445095 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.445075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-g68c4" event={"ID":"895a680f-2d1a-4c04-b9c9-7be40f8cd95c","Type":"ContainerStarted","Data":"34cb25cb5a26916d380225efa5896a11503fd8fed2b39b6d01eadfd126be4134"}
Sep 09 19:56:24.466366 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.466328    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wwrln" podStartSLOduration=65.427639717 podStartE2EDuration="1m12.466314759s" totalImagesPullingTime="7.038675042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:24.465868565 +0000 UTC m=+73.866545758" watchObservedRunningTime="2026-09-09 19:56:24.466314759 +0000 UTC m=+73.866991950"
Sep 09 19:56:24.495304 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:24.495276    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b87587df-pzhtd"]
Sep 09 19:56:24.498728 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:24.498701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eed42f_a908_4013_bbd4_ceaa766a77f4.slice/crio-444e95bb043827ebf4e94e469b9791f4c39559776fa9aa593c439e90128adad0 WatchSource:0}: Error finding container 444e95bb043827ebf4e94e469b9791f4c39559776fa9aa593c439e90128adad0: Status 404 returned error can't find the container with id 444e95bb043827ebf4e94e469b9791f4c39559776fa9aa593c439e90128adad0
Sep 09 19:56:25.451652 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.451613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b87587df-pzhtd" event={"ID":"81eed42f-a908-4013-bbd4-ceaa766a77f4","Type":"ContainerStarted","Data":"66f883917bdba4ca6c53be72c79a7ba8e761a149974eeda9f54c61ff307709a5"}
Sep 09 19:56:25.452055 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.451669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b87587df-pzhtd" event={"ID":"81eed42f-a908-4013-bbd4-ceaa766a77f4","Type":"ContainerStarted","Data":"444e95bb043827ebf4e94e469b9791f4c39559776fa9aa593c439e90128adad0"}
Sep 09 19:56:25.492045 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.491984    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b87587df-pzhtd" podStartSLOduration=2.491964363 podStartE2EDuration="2.491964363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:25.490400822 +0000 UTC m=+74.891078026" watchObservedRunningTime="2026-09-09 19:56:25.491964363 +0000 UTC m=+74.892641556"
Sep 09 19:56:25.568551 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.568514    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:56:25.584154 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.584121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.585202 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.585142    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:56:25.586841 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.586816    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 19:56:25.586965 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.586825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 19:56:25.587430 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.587404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 19:56:25.589025 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.588254    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 19:56:25.589025 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.588258    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qswmf\""
Sep 09 19:56:25.589025 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.588506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 19:56:25.692182 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.691950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvpr\" (UniqueName: \"kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.692182 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.692002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.692182 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.692070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.692182 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.692108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.692182 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.692178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.692589 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.692231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793407 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793407 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793600 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793600 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvpr\" (UniqueName: \"kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793600 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.793600 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.793591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.794450 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.794376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.794583 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.794514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.794921 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.794861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.796632 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.796588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.797097 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.797073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.803334 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.803314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvpr\" (UniqueName: \"kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr\") pod \"console-547bd4d447-p8sgl\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") " pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:25.898404 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:25.898365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:26.462473 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:26.462440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:27.208474 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.208420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:27.210947 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.210922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b99cff65-7625-4001-835b-52fe979042d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkddx\" (UID: \"b99cff65-7625-4001-835b-52fe979042d4\") " pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:27.401781 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.401556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57xxp\""
Sep 09 19:56:27.409623 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.409595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkddx"
Sep 09 19:56:27.472652 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.472621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:56:27.476970 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.476491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq" event={"ID":"74e15636-ee50-4dd7-9497-49684e67f1a6","Type":"ContainerStarted","Data":"0165d49e374e7a727425675e629e1595277e30430dd11f99b6b965bf874b195f"}
Sep 09 19:56:27.477585 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:27.477537    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10ed5f0_35ce_496a_bb7f_8c616c11f666.slice/crio-da1e3be2cd51b1398041eeea7877e19f1331a15fcc9d03f29e008be9cef670a1 WatchSource:0}: Error finding container da1e3be2cd51b1398041eeea7877e19f1331a15fcc9d03f29e008be9cef670a1: Status 404 returned error can't find the container with id da1e3be2cd51b1398041eeea7877e19f1331a15fcc9d03f29e008be9cef670a1
Sep 09 19:56:27.501040 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.500994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ff4f7d4fd-xz6nq" podStartSLOduration=1.6815955420000002 podStartE2EDuration="4.500976116s" totalImagesPullingTime="2.819380574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.500506885 +0000 UTC m=+76.901184080" watchObservedRunningTime="2026-09-09 19:56:27.500976116 +0000 UTC m=+76.901653308"
Sep 09 19:56:27.689308 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:27.689273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkddx"]
Sep 09 19:56:27.692759 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:27.692727    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99cff65_7625_4001_835b_52fe979042d4.slice/crio-0870906ecb65c3e6fcad8ce52bfc4f742cdf5cb74dc6867b4957fd66a34574a3 WatchSource:0}: Error finding container 0870906ecb65c3e6fcad8ce52bfc4f742cdf5cb74dc6867b4957fd66a34574a3: Status 404 returned error can't find the container with id 0870906ecb65c3e6fcad8ce52bfc4f742cdf5cb74dc6867b4957fd66a34574a3
Sep 09 19:56:28.483013 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:28.482969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkddx" event={"ID":"b99cff65-7625-4001-835b-52fe979042d4","Type":"ContainerStarted","Data":"632357a95ee108bc96b0ad4b885cc0e384d95b428a543e680a352b55b182d570"}
Sep 09 19:56:28.483473 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:28.483055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkddx" event={"ID":"b99cff65-7625-4001-835b-52fe979042d4","Type":"ContainerStarted","Data":"0870906ecb65c3e6fcad8ce52bfc4f742cdf5cb74dc6867b4957fd66a34574a3"}
Sep 09 19:56:28.485275 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:28.485204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547bd4d447-p8sgl" event={"ID":"c10ed5f0-35ce-496a-bb7f-8c616c11f666","Type":"ContainerStarted","Data":"da1e3be2cd51b1398041eeea7877e19f1331a15fcc9d03f29e008be9cef670a1"}
Sep 09 19:56:29.491921 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:29.491839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkddx" event={"ID":"b99cff65-7625-4001-835b-52fe979042d4","Type":"ContainerStarted","Data":"65f52d7053f3442ce31f7cf274ee902154dbbdf4c30006e3c88e97fe6a147434"}
Sep 09 19:56:31.503877 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:31.503839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkddx" event={"ID":"b99cff65-7625-4001-835b-52fe979042d4","Type":"ContainerStarted","Data":"881279d7598fe0c1296b638b8587f3a993588d93f227f4f22752696bd38f80c0"}
Sep 09 19:56:31.505860 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:31.505823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547bd4d447-p8sgl" event={"ID":"c10ed5f0-35ce-496a-bb7f-8c616c11f666","Type":"ContainerStarted","Data":"a5bdccdfb1a2c0faf2e101dcc50390f990a1eb50e529ed79ee94953dd2e15456"}
Sep 09 19:56:31.521972 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:31.521923    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vkddx" podStartSLOduration=33.311762291 podStartE2EDuration="36.52190369s" totalImagesPullingTime="3.210141399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:31.520499338 +0000 UTC m=+80.921176546" watchObservedRunningTime="2026-09-09 19:56:31.52190369 +0000 UTC m=+80.922580886"
Sep 09 19:56:31.537543 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:31.537486    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-547bd4d447-p8sgl" podStartSLOduration=2.982561133 podStartE2EDuration="6.537471076s" totalImagesPullingTime="3.554909943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:31.53656273 +0000 UTC m=+80.937239927" watchObservedRunningTime="2026-09-09 19:56:31.537471076 +0000 UTC m=+80.938148267"
Sep 09 19:56:33.114194 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.114163    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:56:33.118323 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.118297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.134074 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.134046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:56:33.135969 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.135945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 19:56:33.264103 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264277 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csmt\" (UniqueName: \"kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.264469 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.264349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365099 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365099 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365312 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365312 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365417 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365417 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.365417 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.365368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csmt\" (UniqueName: \"kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.366061 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.366028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.366173 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.366097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.366173 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.366126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.366623 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.366599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.368000 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.367977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.368379 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.368357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.373752 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.373728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csmt\" (UniqueName: \"kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt\") pod \"console-585bfb4fc5-rv5mr\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") " pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.436186 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.436155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:33.447380 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.447353    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zw5cx"
Sep 09 19:56:33.686201 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.686053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:33.720313 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:33.720127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:56:33.723527 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:33.723502    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38f0af3_6398_42b2_8fa4_e1fc3b6558af.slice/crio-446ef1a3747943cac445ce8dc132636e34924cab276fcacdabbfcb1dd6505db8 WatchSource:0}: Error finding container 446ef1a3747943cac445ce8dc132636e34924cab276fcacdabbfcb1dd6505db8: Status 404 returned error can't find the container with id 446ef1a3747943cac445ce8dc132636e34924cab276fcacdabbfcb1dd6505db8
Sep 09 19:56:34.519614 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:34.519579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bfb4fc5-rv5mr" event={"ID":"b38f0af3-6398-42b2-8fa4-e1fc3b6558af","Type":"ContainerStarted","Data":"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"}
Sep 09 19:56:34.520105 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:34.519632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bfb4fc5-rv5mr" event={"ID":"b38f0af3-6398-42b2-8fa4-e1fc3b6558af","Type":"ContainerStarted","Data":"446ef1a3747943cac445ce8dc132636e34924cab276fcacdabbfcb1dd6505db8"}
Sep 09 19:56:34.542036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:34.541968    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-585bfb4fc5-rv5mr" podStartSLOduration=1.541946765 podStartE2EDuration="1.541946765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:34.540783852 +0000 UTC m=+83.941461046" watchObservedRunningTime="2026-09-09 19:56:34.541946765 +0000 UTC m=+83.942623959"
Sep 09 19:56:35.899521 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:35.899478    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:35.899990 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:35.899562    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:35.905708 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:35.905611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:36.530667 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:36.530638    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:56:37.353398 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.353360    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"]
Sep 09 19:56:37.358232 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.358202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.360799 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.360714    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:56:37.362036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.361856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 19:56:37.362036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.361911    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-slxcx\""
Sep 09 19:56:37.362036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.361921    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:37.362036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.361933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:56:37.362036 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.361994    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 19:56:37.366915 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.366894    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zrmwd"]
Sep 09 19:56:37.370429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.370408    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.370938 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.370917    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"]
Sep 09 19:56:37.373028 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.373008    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nplxr\""
Sep 09 19:56:37.373373 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.373353    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:37.373525 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.373489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:37.373656 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.373610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:37.502967 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.502919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-root\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.502973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7e48b6-9b20-425c-9821-9d11e3649a7b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.503150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-textfile\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-metrics-client-ca\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.503150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-sys\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf8v\" (UniqueName: \"kubernetes.io/projected/9d7e48b6-9b20-425c-9821-9d11e3649a7b-kube-api-access-mvf8v\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-wtmp\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.503400 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.503388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkmh\" (UniqueName: \"kubernetes.io/projected/cef081dd-ded1-4f7c-94b9-7993b1566b2c-kube-api-access-vqkmh\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.603871 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.603785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf8v\" (UniqueName: \"kubernetes.io/projected/9d7e48b6-9b20-425c-9821-9d11e3649a7b-kube-api-access-mvf8v\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.603871 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.603833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.604072 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.603998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-wtmp\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604072 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604168 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkmh\" (UniqueName: \"kubernetes.io/projected/cef081dd-ded1-4f7c-94b9-7993b1566b2c-kube-api-access-vqkmh\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604225 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-root\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604273 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7e48b6-9b20-425c-9821-9d11e3649a7b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.604327 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-textfile\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604327 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-metrics-client-ca\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604434 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-wtmp\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604434 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.604530 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604530 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-sys\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604530 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604782 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-sys\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.604860 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:37.604841    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:37.604914 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:37.604904    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls podName:cef081dd-ded1-4f7c-94b9-7993b1566b2c nodeName:}" failed. No retries permitted until 2026-09-09 19:56:38.104884216 +0000 UTC m=+87.505561389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls") pod "node-exporter-zrmwd" (UID: "cef081dd-ded1-4f7c-94b9-7993b1566b2c") : secret "node-exporter-tls" not found
Sep 09 19:56:37.604983 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cef081dd-ded1-4f7c-94b9-7993b1566b2c-root\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.605037 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.604976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-textfile\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.605228 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.605204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.605526 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.605483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cef081dd-ded1-4f7c-94b9-7993b1566b2c-metrics-client-ca\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.605642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.605617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7e48b6-9b20-425c-9821-9d11e3649a7b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.606708 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.606654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.607016 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.606991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d7e48b6-9b20-425c-9821-9d11e3649a7b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.607450 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.607431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.612642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.612620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf8v\" (UniqueName: \"kubernetes.io/projected/9d7e48b6-9b20-425c-9821-9d11e3649a7b-kube-api-access-mvf8v\") pod \"openshift-state-metrics-657fd6c54-s92ft\" (UID: \"9d7e48b6-9b20-425c-9821-9d11e3649a7b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:37.613121 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.613100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkmh\" (UniqueName: \"kubernetes.io/projected/cef081dd-ded1-4f7c-94b9-7993b1566b2c-kube-api-access-vqkmh\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:37.669840 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:37.669810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"
Sep 09 19:56:38.110300 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.110257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:38.112783 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.112760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cef081dd-ded1-4f7c-94b9-7993b1566b2c-node-exporter-tls\") pod \"node-exporter-zrmwd\" (UID: \"cef081dd-ded1-4f7c-94b9-7993b1566b2c\") " pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:38.282326 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.282288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zrmwd"
Sep 09 19:56:38.454909 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.454878    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:56:38.458694 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.458657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.461358 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461335    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pwjjc\""
Sep 09 19:56:38.461534 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461484    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 19:56:38.461611 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461559    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 19:56:38.461611 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 19:56:38.461611 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 19:56:38.461804 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 19:56:38.461804 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461576    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 19:56:38.462002 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.461983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 19:56:38.462148 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.462001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 19:56:38.462237 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.462010    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 19:56:38.477883 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.477863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:56:38.616218 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616399 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616399 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616399 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-out\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616399 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjst5\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-kube-api-access-xjst5\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616867 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-web-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616867 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.616867 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.616727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717517 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717517 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjst5\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-kube-api-access-xjst5\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-web-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.717778 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.718228 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.718228 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-out\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.718228 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.717926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.719058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.718730    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.719058 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.718994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.720873 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.720846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-out\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.721033 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.721009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.721499 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.721383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722260 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722260 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722455 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722570 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722901 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.722989 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.722946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-web-config\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.726540 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.726515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjst5\" (UniqueName: \"kubernetes.io/projected/ac1d7ef2-c581-4ee3-816b-8e14e50d9113-kube-api-access-xjst5\") pod \"alertmanager-main-0\" (UID: \"ac1d7ef2-c581-4ee3-816b-8e14e50d9113\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:38.769144 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:38.769113    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 19:56:40.366505 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.366470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"]
Sep 09 19:56:40.370297 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.370273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.372960 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.372849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fglww\""
Sep 09 19:56:40.373088 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.372964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6i4aiiet4etnm\""
Sep 09 19:56:40.373088 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.373060    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 19:56:40.373215 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.373124    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 19:56:40.373549 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.373527    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 19:56:40.373549 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.373544    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 19:56:40.373711 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.373548    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 19:56:40.381299 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.381277    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"]
Sep 09 19:56:40.533654 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.533840 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.533840 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885160f-0aa1-4e01-b64a-194e61e0c276-metrics-client-ca\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.533939 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.533939 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfdw\" (UniqueName: \"kubernetes.io/projected/2885160f-0aa1-4e01-b64a-194e61e0c276-kube-api-access-tpfdw\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.533939 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-grpc-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.534071 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.534071 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.533982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635485 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635485 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885160f-0aa1-4e01-b64a-194e61e0c276-metrics-client-ca\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfdw\" (UniqueName: \"kubernetes.io/projected/2885160f-0aa1-4e01-b64a-194e61e0c276-kube-api-access-tpfdw\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.635721 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.635659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-grpc-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.636476 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.636418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2885160f-0aa1-4e01-b64a-194e61e0c276-metrics-client-ca\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.638701 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.638663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.638822 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.638792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-grpc-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.639168 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.639144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.639257 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.639154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-tls\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.639257 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.639199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.639477 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.639449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2885160f-0aa1-4e01-b64a-194e61e0c276-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.645916 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.645895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfdw\" (UniqueName: \"kubernetes.io/projected/2885160f-0aa1-4e01-b64a-194e61e0c276-kube-api-access-tpfdw\") pod \"thanos-querier-754fb6dd44-bmlxn\" (UID: \"2885160f-0aa1-4e01-b64a-194e61e0c276\") " pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:40.682885 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:40.682858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:42.334131 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:42.334095    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:56:42.480398 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:42.480127    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef081dd_ded1_4f7c_94b9_7993b1566b2c.slice/crio-b8f25e422b46fe36bb1cdb7c38f33f81e04b9aa835b12c2132d847cf099e4a3b WatchSource:0}: Error finding container b8f25e422b46fe36bb1cdb7c38f33f81e04b9aa835b12c2132d847cf099e4a3b: Status 404 returned error can't find the container with id b8f25e422b46fe36bb1cdb7c38f33f81e04b9aa835b12c2132d847cf099e4a3b
Sep 09 19:56:42.552809 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:42.552777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zrmwd" event={"ID":"cef081dd-ded1-4f7c-94b9-7993b1566b2c","Type":"ContainerStarted","Data":"b8f25e422b46fe36bb1cdb7c38f33f81e04b9aa835b12c2132d847cf099e4a3b"}
Sep 09 19:56:42.757142 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:42.757118    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft"]
Sep 09 19:56:42.764313 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:42.764283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"]
Sep 09 19:56:42.765629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:42.765605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 19:56:42.782574 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:42.782540    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7e48b6_9b20_425c_9821_9d11e3649a7b.slice/crio-0e88ed42014fc7492644815a9cff5dac372e1770cc7c7bbde02475d326051d3e WatchSource:0}: Error finding container 0e88ed42014fc7492644815a9cff5dac372e1770cc7c7bbde02475d326051d3e: Status 404 returned error can't find the container with id 0e88ed42014fc7492644815a9cff5dac372e1770cc7c7bbde02475d326051d3e
Sep 09 19:56:42.783675 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:42.783650    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2885160f_0aa1_4e01_b64a_194e61e0c276.slice/crio-f28727f72c4a4db705c50addaae5a08a0643d94b8f51bdbe39ed567d91ed3773 WatchSource:0}: Error finding container f28727f72c4a4db705c50addaae5a08a0643d94b8f51bdbe39ed567d91ed3773: Status 404 returned error can't find the container with id f28727f72c4a4db705c50addaae5a08a0643d94b8f51bdbe39ed567d91ed3773
Sep 09 19:56:42.784483 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:56:42.784464    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1d7ef2_c581_4ee3_816b_8e14e50d9113.slice/crio-1e47b2194134b7a5b0cccb29b94521df431577015cdb850cc42bc1d3d4930e64 WatchSource:0}: Error finding container 1e47b2194134b7a5b0cccb29b94521df431577015cdb850cc42bc1d3d4930e64: Status 404 returned error can't find the container with id 1e47b2194134b7a5b0cccb29b94521df431577015cdb850cc42bc1d3d4930e64
Sep 09 19:56:43.436966 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.436934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:56:43.562409 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.561924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"f28727f72c4a4db705c50addaae5a08a0643d94b8f51bdbe39ed567d91ed3773"}
Sep 09 19:56:43.566531 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.566401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft" event={"ID":"9d7e48b6-9b20-425c-9821-9d11e3649a7b","Type":"ContainerStarted","Data":"8e9345f7c2eb3fbeb84f38cf655a5fb6905dfca05a1c11a574f17497cbf5ea6e"}
Sep 09 19:56:43.566531 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.566464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft" event={"ID":"9d7e48b6-9b20-425c-9821-9d11e3649a7b","Type":"ContainerStarted","Data":"79b6db856673f6c06f567b9127c04f5d0ea6761742833ad3cbbfd867ecce60a8"}
Sep 09 19:56:43.566531 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.566492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft" event={"ID":"9d7e48b6-9b20-425c-9821-9d11e3649a7b","Type":"ContainerStarted","Data":"0e88ed42014fc7492644815a9cff5dac372e1770cc7c7bbde02475d326051d3e"}
Sep 09 19:56:43.568697 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.568530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zrmwd" event={"ID":"cef081dd-ded1-4f7c-94b9-7993b1566b2c","Type":"ContainerStarted","Data":"4a72001bb81a4efe793b2dcb69c279e02c966a85444a05748e1568d39d7d085d"}
Sep 09 19:56:43.572934 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.572820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"1e47b2194134b7a5b0cccb29b94521df431577015cdb850cc42bc1d3d4930e64"}
Sep 09 19:56:43.575110 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.574991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-g68c4" event={"ID":"895a680f-2d1a-4c04-b9c9-7be40f8cd95c","Type":"ContainerStarted","Data":"a17f0e09c333b7d322b83b8caab75978a7575e534be4fbd4570fa4f3738c63dc"}
Sep 09 19:56:43.575663 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.575635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:43.582777 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.582749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-g68c4"
Sep 09 19:56:43.611987 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:43.611931    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-g68c4" podStartSLOduration=2.17214363 podStartE2EDuration="20.611914495s" totalImagesPullingTime="18.439770865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:43.609606652 +0000 UTC m=+93.010283865" watchObservedRunningTime="2026-09-09 19:56:43.611914495 +0000 UTC m=+93.012591688"
Sep 09 19:56:44.581179 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:44.581135    2582 generic.go:356] "Generic (PLEG): container finished" podID="cef081dd-ded1-4f7c-94b9-7993b1566b2c" containerID="4a72001bb81a4efe793b2dcb69c279e02c966a85444a05748e1568d39d7d085d" exitCode=0
Sep 09 19:56:44.581774 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:44.581221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zrmwd" event={"ID":"cef081dd-ded1-4f7c-94b9-7993b1566b2c","Type":"ContainerDied","Data":"4a72001bb81a4efe793b2dcb69c279e02c966a85444a05748e1568d39d7d085d"}
Sep 09 19:56:45.588885 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.588774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft" event={"ID":"9d7e48b6-9b20-425c-9821-9d11e3649a7b","Type":"ContainerStarted","Data":"c778378405c32528b5872f16ee0c01537e1cfef108782fc95fcf437f15029ce9"}
Sep 09 19:56:45.591639 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.591608    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zrmwd" event={"ID":"cef081dd-ded1-4f7c-94b9-7993b1566b2c","Type":"ContainerStarted","Data":"2b91bf9bb9951579b98ef4632dbb4eff9306e9f6e6c0cf85ac80ad2b89cec0f7"}
Sep 09 19:56:45.591790 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.591662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zrmwd" event={"ID":"cef081dd-ded1-4f7c-94b9-7993b1566b2c","Type":"ContainerStarted","Data":"c00ef8f14b8fec43b0d47d93c07b4421345a0fa178fac9bd0a311023d1da98fb"}
Sep 09 19:56:45.594334 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.594303    2582 generic.go:356] "Generic (PLEG): container finished" podID="ac1d7ef2-c581-4ee3-816b-8e14e50d9113" containerID="7134481dad4361855035d2b894e1835907edb5f2207aa73efd57e3392e98ba97" exitCode=0
Sep 09 19:56:45.594462 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.594406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerDied","Data":"7134481dad4361855035d2b894e1835907edb5f2207aa73efd57e3392e98ba97"}
Sep 09 19:56:45.612020 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.610255    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-s92ft" podStartSLOduration=6.515273443 podStartE2EDuration="8.610236672s" totalImagesPullingTime="2.094963229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:45.609533674 +0000 UTC m=+95.010210879" watchObservedRunningTime="2026-09-09 19:56:45.610236672 +0000 UTC m=+95.010913865"
Sep 09 19:56:45.661327 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:45.661267    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zrmwd" podStartSLOduration=7.648179968 podStartE2EDuration="8.661246215s" totalImagesPullingTime="934.387931ms" totalInitContainerRuntime="78.678316ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:45.659219587 +0000 UTC m=+95.059896782" watchObservedRunningTime="2026-09-09 19:56:45.661246215 +0000 UTC m=+95.061923408"
Sep 09 19:56:47.483256 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:47.483223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b87587df-pzhtd"
Sep 09 19:56:47.606823 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:47.606726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"9689bc522bcd75e28510c5c72a0d43002f04a14f3821990d45d37e6c5462bcb3"}
Sep 09 19:56:47.606823 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:47.606819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"7c8473627b3bb9ff7419b9058afb97abb6054f7f124d2c5fde60f1b4f759654e"}
Sep 09 19:56:47.607029 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:47.606861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"6ec6618e36ae976f003b9f8c73ddcf6914f41333fed94d12b799e489918678cc"}
Sep 09 19:56:48.704040 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:48.703990    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerName="registry" containerID="cri-o://be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1" gracePeriod=30
Sep 09 19:56:48.977039 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:48.977012    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:49.019652 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.019793 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.019793 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019762    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019846    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmct2\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.019913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") pod \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\" (UID: \"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90\") "
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.020198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:49.020926 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.020517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:49.024340 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.024236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.024527 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.024493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.026546 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.026500    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:49.027198 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.027163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2" (OuterVolumeSpecName: "kube-api-access-kmct2") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "kube-api-access-kmct2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.027198 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.027174    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:49.031405 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.031377    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" (UID: "6b226cb1-db1d-4928-98bd-2cc6ebcd4c90"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:56:49.121189 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121164    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-ca-trust-extracted\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121189 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121190    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-certificates\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121203    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-bound-sa-token\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121217    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-image-registry-private-configuration\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121227    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-trusted-ca\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121236    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-installation-pull-secrets\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121250    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmct2\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-kube-api-access-kmct2\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.121348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.121260    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90-registry-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.626429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.625130    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerID="be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1" exitCode=0
Sep 09 19:56:49.626429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.625232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" event={"ID":"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90","Type":"ContainerDied","Data":"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"}
Sep 09 19:56:49.626429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.625314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw" event={"ID":"6b226cb1-db1d-4928-98bd-2cc6ebcd4c90","Type":"ContainerDied","Data":"73c6e3916ed809067e2fb6ddf78fa64b9c42b9bdd0eec6c61579aba63d0aa762"}
Sep 09 19:56:49.626429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.625326    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfbc545f6-jjpvw"
Sep 09 19:56:49.626429 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.625348    2582 scope.go:117] "RemoveContainer" containerID="be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"
Sep 09 19:56:49.632149 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.632126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"4e5ec1f6b4cdf3a521f9188894bb2b379d7f9cdcf7d3fbd05610f69363ae3ac5"}
Sep 09 19:56:49.632268 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.632196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"5bdc47a3131ec663d79b3fca188f81788159fa68c11483a1aa4aeeee8b70f553"}
Sep 09 19:56:49.632268 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.632240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"8a0a64c10d2e7d7ea6fd0fb8698630d73fcd9196c6228ba240f85e3737b56e0b"}
Sep 09 19:56:49.632376 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.632277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"a4c1da2081f54242f113b7bc1fabf7515dcc14921f46fc21997cf5af016beaba"}
Sep 09 19:56:49.632376 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.632317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"e4c7ab6148411e8461fc8154924c86aa674a3805f2c6d635f09e97a112319b2f"}
Sep 09 19:56:49.638661 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.638640    2582 scope.go:117] "RemoveContainer" containerID="be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"
Sep 09 19:56:49.639295 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:56:49.639267    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1\": container with ID starting with be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1 not found: ID does not exist" containerID="be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"
Sep 09 19:56:49.639533 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.639304    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1"} err="failed to get container status \"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1\": rpc error: code = NotFound desc = could not find container \"be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1\": container with ID starting with be513cb049eff0366a574881ec6bfb0aeb8a8c26e772c18530809ca495d8d1d1 not found: ID does not exist"
Sep 09 19:56:49.643890 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.643855    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:56:49.648470 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:49.648447    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dfbc545f6-jjpvw"]
Sep 09 19:56:51.127175 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.127139    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" path="/var/lib/kubelet/pods/6b226cb1-db1d-4928-98bd-2cc6ebcd4c90/volumes"
Sep 09 19:56:51.647032 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.646990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac1d7ef2-c581-4ee3-816b-8e14e50d9113","Type":"ContainerStarted","Data":"52bc69510e0048ba916c37177e322fd572a2e3b0b3a272b2e99f5c11c67f8819"}
Sep 09 19:56:51.650718 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.650669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"b3853e17acfb02457e53bedc8d749fb2955d169b0cdb4e6c38f89c3a76120f9e"}
Sep 09 19:56:51.650860 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.650756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"573ccb14c65e344f7ce8022af4f0c79672d904c36c3f0bbdb489578a17678152"}
Sep 09 19:56:51.650860 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.650800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" event={"ID":"2885160f-0aa1-4e01-b64a-194e61e0c276","Type":"ContainerStarted","Data":"39ca17e55a94ef8451cfc186766235b68302ea9ebf6dd7faf718124f821f5d0a"}
Sep 09 19:56:51.650860 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.650849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:51.674692 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.674635    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.120488432 podStartE2EDuration="13.67461398s" totalImagesPullingTime="6.530420811s" totalInitContainerRuntime="23.704737ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:51.672695242 +0000 UTC m=+101.073372433" watchObservedRunningTime="2026-09-09 19:56:51.67461398 +0000 UTC m=+101.075291173"
Sep 09 19:56:51.696056 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:51.695736    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn" podStartSLOduration=4.369894198 podStartE2EDuration="11.695718416s" totalImagesPullingTime="7.325824218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:51.695257435 +0000 UTC m=+101.095934630" watchObservedRunningTime="2026-09-09 19:56:51.695718416 +0000 UTC m=+101.096395608"
Sep 09 19:56:52.436711 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:52.436663    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xqdk9"
Sep 09 19:56:52.660652 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:52.660621    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-754fb6dd44-bmlxn"
Sep 09 19:56:53.622543 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:56:53.622513    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:57:04.701889 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:04.701857    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b080784-3270-422d-bced-001ff0632258" containerID="55f9983d52941c71485aa55f76aab4648ee6cc6829278825b88c379c89e12fe2" exitCode=0
Sep 09 19:57:04.702273 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:04.701929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8" event={"ID":"9b080784-3270-422d-bced-001ff0632258","Type":"ContainerDied","Data":"55f9983d52941c71485aa55f76aab4648ee6cc6829278825b88c379c89e12fe2"}
Sep 09 19:57:04.702273 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:04.702253    2582 scope.go:117] "RemoveContainer" containerID="55f9983d52941c71485aa55f76aab4648ee6cc6829278825b88c379c89e12fe2"
Sep 09 19:57:05.708667 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:05.708637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-r7ll8" event={"ID":"9b080784-3270-422d-bced-001ff0632258","Type":"ContainerStarted","Data":"40f41a5a05b2b7592b0a4191ad6f88ca7c4614e565dc66f61f64e96e66c64679"}
Sep 09 19:57:07.356063 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.356020    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-585bfb4fc5-rv5mr" podUID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" containerName="console" containerID="cri-o://0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc" gracePeriod=15
Sep 09 19:57:07.632258 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.632200    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:57:07.666616 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666767 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666767 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666767 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666897 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666897 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666836    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4csmt\" (UniqueName: \"kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.666897 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.666864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config\") pod \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\" (UID: \"b38f0af3-6398-42b2-8fa4-e1fc3b6558af\") "
Sep 09 19:57:07.667133 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.667018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca" (OuterVolumeSpecName: "service-ca") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:07.667329 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.667298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:07.667420 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.667324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config" (OuterVolumeSpecName: "console-config") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:07.667420 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.667308    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:07.669217 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.669192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:07.669217 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.669212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:07.669380 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.669310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt" (OuterVolumeSpecName: "kube-api-access-4csmt") pod "b38f0af3-6398-42b2-8fa4-e1fc3b6558af" (UID: "b38f0af3-6398-42b2-8fa4-e1fc3b6558af"). InnerVolumeSpecName "kube-api-access-4csmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:07.717229 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.717196    2582 generic.go:356] "Generic (PLEG): container finished" podID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" containerID="0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc" exitCode=0
Sep 09 19:57:07.717321 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.717235    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585bfb4fc5-rv5mr"
Sep 09 19:57:07.717321 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.717237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bfb4fc5-rv5mr" event={"ID":"b38f0af3-6398-42b2-8fa4-e1fc3b6558af","Type":"ContainerDied","Data":"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"}
Sep 09 19:57:07.717321 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.717290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585bfb4fc5-rv5mr" event={"ID":"b38f0af3-6398-42b2-8fa4-e1fc3b6558af","Type":"ContainerDied","Data":"446ef1a3747943cac445ce8dc132636e34924cab276fcacdabbfcb1dd6505db8"}
Sep 09 19:57:07.717321 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.717310    2582 scope.go:117] "RemoveContainer" containerID="0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"
Sep 09 19:57:07.725735 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.725716    2582 scope.go:117] "RemoveContainer" containerID="0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"
Sep 09 19:57:07.725984 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:57:07.725967    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc\": container with ID starting with 0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc not found: ID does not exist" containerID="0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"
Sep 09 19:57:07.726044 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.725993    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc"} err="failed to get container status \"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc\": rpc error: code = NotFound desc = could not find container \"0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc\": container with ID starting with 0e30ce2fbd3482019983dcfaaa0506a381a9d0066b1d5da3ae92641287bb07dc not found: ID does not exist"
Sep 09 19:57:07.737924 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.737899    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:57:07.742554 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.742535    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-585bfb4fc5-rv5mr"]
Sep 09 19:57:07.768168 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768144    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-trusted-ca-bundle\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768168 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768166    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-oauth-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768177    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csmt\" (UniqueName: \"kubernetes.io/projected/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-kube-api-access-4csmt\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768187    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768196    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-service-ca\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768203    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:07.768303 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:07.768212    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b38f0af3-6398-42b2-8fa4-e1fc3b6558af-console-oauth-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:09.123805 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:09.123764    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" path="/var/lib/kubelet/pods/b38f0af3-6398-42b2-8fa4-e1fc3b6558af/volumes"
Sep 09 19:57:18.641827 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:18.641780    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-547bd4d447-p8sgl" podUID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" containerName="console" containerID="cri-o://a5bdccdfb1a2c0faf2e101dcc50390f990a1eb50e529ed79ee94953dd2e15456" gracePeriod=15
Sep 09 19:57:18.761403 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:18.761367    2582 generic.go:356] "Generic (PLEG): container finished" podID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" containerID="a5bdccdfb1a2c0faf2e101dcc50390f990a1eb50e529ed79ee94953dd2e15456" exitCode=0
Sep 09 19:57:18.761571 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:18.761423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547bd4d447-p8sgl" event={"ID":"c10ed5f0-35ce-496a-bb7f-8c616c11f666","Type":"ContainerDied","Data":"a5bdccdfb1a2c0faf2e101dcc50390f990a1eb50e529ed79ee94953dd2e15456"}
Sep 09 19:57:18.935602 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:18.935238    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:57:19.062224 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062194    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kvpr\" (UniqueName: \"kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062224 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062475 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062270    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062475 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062302    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062475 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062475 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062367    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert\") pod \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\" (UID: \"c10ed5f0-35ce-496a-bb7f-8c616c11f666\") "
Sep 09 19:57:19.062779 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config" (OuterVolumeSpecName: "console-config") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:19.062779 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca" (OuterVolumeSpecName: "service-ca") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:19.062927 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.062786    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:19.064463 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.064437    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:19.064550 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.064532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr" (OuterVolumeSpecName: "kube-api-access-2kvpr") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "kube-api-access-2kvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:19.064735 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.064713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c10ed5f0-35ce-496a-bb7f-8c616c11f666" (UID: "c10ed5f0-35ce-496a-bb7f-8c616c11f666"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163265    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kvpr\" (UniqueName: \"kubernetes.io/projected/c10ed5f0-35ce-496a-bb7f-8c616c11f666-kube-api-access-2kvpr\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163292    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-oauth-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163302    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163311    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-service-ca\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163320    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-console-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.163335 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.163329    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c10ed5f0-35ce-496a-bb7f-8c616c11f666-oauth-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:57:19.775496 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.775462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547bd4d447-p8sgl" event={"ID":"c10ed5f0-35ce-496a-bb7f-8c616c11f666","Type":"ContainerDied","Data":"da1e3be2cd51b1398041eeea7877e19f1331a15fcc9d03f29e008be9cef670a1"}
Sep 09 19:57:19.776002 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.775543    2582 scope.go:117] "RemoveContainer" containerID="a5bdccdfb1a2c0faf2e101dcc50390f990a1eb50e529ed79ee94953dd2e15456"
Sep 09 19:57:19.776002 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.775468    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547bd4d447-p8sgl"
Sep 09 19:57:19.791743 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.791720    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:57:19.795470 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:19.795450    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-547bd4d447-p8sgl"]
Sep 09 19:57:21.123905 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:57:21.123874    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" path="/var/lib/kubelet/pods/c10ed5f0-35ce-496a-bb7f-8c616c11f666/volumes"
Sep 09 19:58:01.731621 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731587    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"]
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731872    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" containerName="console"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731883    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" containerName="console"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731891    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerName="registry"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731897    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerName="registry"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731923    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" containerName="console"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731928    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" containerName="console"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731968    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10ed5f0-35ce-496a-bb7f-8c616c11f666" containerName="console"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731979    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b226cb1-db1d-4928-98bd-2cc6ebcd4c90" containerName="registry"
Sep 09 19:58:01.732068 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.731986    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38f0af3-6398-42b2-8fa4-e1fc3b6558af" containerName="console"
Sep 09 19:58:01.734741 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.734723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.738919 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.738895    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 19:58:01.739391 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.739376    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 19:58:01.739464 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.739434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 19:58:01.739757 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.739735    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zhn6x\""
Sep 09 19:58:01.739897 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.739880    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 19:58:01.740459 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.740440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 19:58:01.750110 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.748945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 19:58:01.753798 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.753776    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"]
Sep 09 19:58:01.792613 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-federate-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792752 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792752 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792752 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-serving-certs-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792880 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-metrics-client-ca\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792880 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792880 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfl4v\" (UniqueName: \"kubernetes.io/projected/d6e63ebe-da2c-4833-aa6f-522980588a96-kube-api-access-wfl4v\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.792880 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.792856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.893907 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.893862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-metrics-client-ca\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.893907 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.893917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894101 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.893938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfl4v\" (UniqueName: \"kubernetes.io/projected/d6e63ebe-da2c-4833-aa6f-522980588a96-kube-api-access-wfl4v\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894101 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.893963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894101 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.893989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-federate-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894101 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.894015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894101 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.894050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894348 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.894104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-serving-certs-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.894836 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.894809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-serving-certs-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.895019 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.894810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-metrics-client-ca\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.895382 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.895356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.896537 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.896513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-federate-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.896625 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.896562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-telemeter-client-tls\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.896671 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.896628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.896856 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.896832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d6e63ebe-da2c-4833-aa6f-522980588a96-secret-telemeter-client\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:01.902536 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:01.902517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfl4v\" (UniqueName: \"kubernetes.io/projected/d6e63ebe-da2c-4833-aa6f-522980588a96-kube-api-access-wfl4v\") pod \"telemeter-client-5d44f64c6-pwnwg\" (UID: \"d6e63ebe-da2c-4833-aa6f-522980588a96\") " pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:02.043871 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:02.043782    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"
Sep 09 19:58:02.323472 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:02.323381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d44f64c6-pwnwg"]
Sep 09 19:58:02.326457 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:58:02.326431    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e63ebe_da2c_4833_aa6f_522980588a96.slice/crio-35d0889334768e05c741708afbb0499ed3df4cdccf2c7a25d05ad028c84fbb64 WatchSource:0}: Error finding container 35d0889334768e05c741708afbb0499ed3df4cdccf2c7a25d05ad028c84fbb64: Status 404 returned error can't find the container with id 35d0889334768e05c741708afbb0499ed3df4cdccf2c7a25d05ad028c84fbb64
Sep 09 19:58:02.942980 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:02.942940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg" event={"ID":"d6e63ebe-da2c-4833-aa6f-522980588a96","Type":"ContainerStarted","Data":"35d0889334768e05c741708afbb0499ed3df4cdccf2c7a25d05ad028c84fbb64"}
Sep 09 19:58:03.948071 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:03.948033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg" event={"ID":"d6e63ebe-da2c-4833-aa6f-522980588a96","Type":"ContainerStarted","Data":"7a6b6f80429bf2dfb80c86b2db3c1213d200be4a93efd0cf02ed84729510fc80"}
Sep 09 19:58:04.954065 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:04.954028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg" event={"ID":"d6e63ebe-da2c-4833-aa6f-522980588a96","Type":"ContainerStarted","Data":"ef265fa4e63007f98f61bd6e492934cf3312d529b45c80b50a6405ba236e2f8f"}
Sep 09 19:58:04.954410 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:04.954090    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg" event={"ID":"d6e63ebe-da2c-4833-aa6f-522980588a96","Type":"ContainerStarted","Data":"2015655402f3719a28e870d02ab70c406fbe4e84d5d623ba35d3385bc8b64653"}
Sep 09 19:58:04.979419 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:04.979368    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d44f64c6-pwnwg" podStartSLOduration=2.423205408 podStartE2EDuration="3.979353197s" totalImagesPullingTime="1.556147789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:04.977934221 +0000 UTC m=+174.378611416" watchObservedRunningTime="2026-09-09 19:58:04.979353197 +0000 UTC m=+174.380030388"
Sep 09 19:58:05.084150 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.084112    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:05.086137 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.086121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.088609 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.088561    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 19:58:05.088826 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.088779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qswmf\""
Sep 09 19:58:05.088918 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.088901    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 19:58:05.088975 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.088935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 19:58:05.088975 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.088958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 19:58:05.089064 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.089000    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 19:58:05.095048 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.095030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 19:58:05.097603 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.097578    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:05.124405 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124552 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124737 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124737 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.124737 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.124668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwf8\" (UniqueName: \"kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.225653 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.225583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.225653 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.225616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.225856 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.225833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.225946 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.225922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226080 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226148 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226148 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwf8\" (UniqueName: \"kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226484 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226597 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.226597 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.226583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.227129 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.227112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.228722 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.228702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.228782 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.228711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.237449 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.237425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwf8\" (UniqueName: \"kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8\") pod \"console-588466fb46-xp7jm\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") " pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.395669 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.395632    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:05.667374 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.667281    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:05.673516 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:58:05.673478    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5801c08_63ca_4948_9d32_81a5fcb2195d.slice/crio-6eae0e037acd8dcaf546e0efec2ddb98f0bd425666f338d3393c57d61000c2b3 WatchSource:0}: Error finding container 6eae0e037acd8dcaf546e0efec2ddb98f0bd425666f338d3393c57d61000c2b3: Status 404 returned error can't find the container with id 6eae0e037acd8dcaf546e0efec2ddb98f0bd425666f338d3393c57d61000c2b3
Sep 09 19:58:05.959756 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.959718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588466fb46-xp7jm" event={"ID":"e5801c08-63ca-4948-9d32-81a5fcb2195d","Type":"ContainerStarted","Data":"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"}
Sep 09 19:58:05.960138 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.959775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588466fb46-xp7jm" event={"ID":"e5801c08-63ca-4948-9d32-81a5fcb2195d","Type":"ContainerStarted","Data":"6eae0e037acd8dcaf546e0efec2ddb98f0bd425666f338d3393c57d61000c2b3"}
Sep 09 19:58:05.979912 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:05.979866    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-588466fb46-xp7jm" podStartSLOduration=0.979847324 podStartE2EDuration="979.847324ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:05.978431411 +0000 UTC m=+175.379108613" watchObservedRunningTime="2026-09-09 19:58:05.979847324 +0000 UTC m=+175.380524517"
Sep 09 19:58:15.396217 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.396183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:15.396602 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.396251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:15.400945 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.400924    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:15.432747 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.432722    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:15.461410 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.461384    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 19:58:15.464131 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.464115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.475302 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.475274    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 19:58:15.507338 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507427 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507427 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz27t\" (UniqueName: \"kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507514 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507676 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507769 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.507769 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.507759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608139 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608302 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608302 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608302 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608467 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608467 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608467 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz27t\" (UniqueName: \"kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.608928 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.608904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.609069 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.609048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.609167 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.609140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.609205 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.609152    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.610629 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.610609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.610832 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.610814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.616317 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.616298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz27t\" (UniqueName: \"kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t\") pod \"console-858d6458c8-bkk52\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") " pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:15.773485 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:15.773452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:16.003829 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:16.003799    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:16.048190 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:16.048119    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 19:58:16.051404 ip-10-0-135-16 kubenswrapper[2582]: W0909 19:58:16.051373    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3804f434_d2a4_4fce_9e3c_5bb49b0ccf0f.slice/crio-f603998ae071b3dc8e0d38370fe6332132c3b32bf1b9fed1c76dbdeaf85f9bae WatchSource:0}: Error finding container f603998ae071b3dc8e0d38370fe6332132c3b32bf1b9fed1c76dbdeaf85f9bae: Status 404 returned error can't find the container with id f603998ae071b3dc8e0d38370fe6332132c3b32bf1b9fed1c76dbdeaf85f9bae
Sep 09 19:58:17.008019 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:17.007983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858d6458c8-bkk52" event={"ID":"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f","Type":"ContainerStarted","Data":"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"}
Sep 09 19:58:17.008424 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:17.008038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858d6458c8-bkk52" event={"ID":"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f","Type":"ContainerStarted","Data":"f603998ae071b3dc8e0d38370fe6332132c3b32bf1b9fed1c76dbdeaf85f9bae"}
Sep 09 19:58:17.029213 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:17.029170    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858d6458c8-bkk52" podStartSLOduration=2.029156308 podStartE2EDuration="2.029156308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:17.02803199 +0000 UTC m=+186.428709182" watchObservedRunningTime="2026-09-09 19:58:17.029156308 +0000 UTC m=+186.429833502"
Sep 09 19:58:25.773807 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:25.773752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:25.773807 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:25.773828    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:25.778744 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:25.778720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:26.045750 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:26.045664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 19:58:42.026785 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.026715    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-588466fb46-xp7jm" podUID="e5801c08-63ca-4948-9d32-81a5fcb2195d" containerName="console" containerID="cri-o://03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c" gracePeriod=15
Sep 09 19:58:42.290219 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.290195    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:42.332501 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332540    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332642 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332821 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332670    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drwf8\" (UniqueName: \"kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332821 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config\") pod \"e5801c08-63ca-4948-9d32-81a5fcb2195d\" (UID: \"e5801c08-63ca-4948-9d32-81a5fcb2195d\") "
Sep 09 19:58:42.332989 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.332911    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:42.333114 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.333092    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-oauth-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.333180 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.333140    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:42.333264 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.333246    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config" (OuterVolumeSpecName: "console-config") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:42.333264 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.333244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca" (OuterVolumeSpecName: "service-ca") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:42.334714 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.334670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:42.334795 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.334745    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:42.334795 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.334751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8" (OuterVolumeSpecName: "kube-api-access-drwf8") pod "e5801c08-63ca-4948-9d32-81a5fcb2195d" (UID: "e5801c08-63ca-4948-9d32-81a5fcb2195d"). InnerVolumeSpecName "kube-api-access-drwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:58:42.434214 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434178    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-trusted-ca-bundle\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.434214 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434213    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-service-ca\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.434214 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434223    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwf8\" (UniqueName: \"kubernetes.io/projected/e5801c08-63ca-4948-9d32-81a5fcb2195d-kube-api-access-drwf8\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.434388 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434233    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.434388 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434241    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:42.434388 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:42.434249    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5801c08-63ca-4948-9d32-81a5fcb2195d-console-oauth-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 19:58:43.105151 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.105111    2582 generic.go:356] "Generic (PLEG): container finished" podID="e5801c08-63ca-4948-9d32-81a5fcb2195d" containerID="03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c" exitCode=0
Sep 09 19:58:43.105151 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.105151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588466fb46-xp7jm"
Sep 09 19:58:43.105596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.105198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588466fb46-xp7jm" event={"ID":"e5801c08-63ca-4948-9d32-81a5fcb2195d","Type":"ContainerDied","Data":"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"}
Sep 09 19:58:43.105596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.105256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588466fb46-xp7jm" event={"ID":"e5801c08-63ca-4948-9d32-81a5fcb2195d","Type":"ContainerDied","Data":"6eae0e037acd8dcaf546e0efec2ddb98f0bd425666f338d3393c57d61000c2b3"}
Sep 09 19:58:43.105596 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.105278    2582 scope.go:117] "RemoveContainer" containerID="03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"
Sep 09 19:58:43.113870 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.113853    2582 scope.go:117] "RemoveContainer" containerID="03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"
Sep 09 19:58:43.114131 ip-10-0-135-16 kubenswrapper[2582]: E0909 19:58:43.114114    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c\": container with ID starting with 03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c not found: ID does not exist" containerID="03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"
Sep 09 19:58:43.114190 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.114143    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c"} err="failed to get container status \"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c\": rpc error: code = NotFound desc = could not find container \"03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c\": container with ID starting with 03d8fae824711b515081627f36fed0f732fa56f3d51145dc2972a32e858d003c not found: ID does not exist"
Sep 09 19:58:43.126055 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.126032    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:43.129822 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:43.129803    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-588466fb46-xp7jm"]
Sep 09 19:58:45.123247 ip-10-0-135-16 kubenswrapper[2582]: I0909 19:58:45.123210    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5801c08-63ca-4948-9d32-81a5fcb2195d" path="/var/lib/kubelet/pods/e5801c08-63ca-4948-9d32-81a5fcb2195d/volumes"
Sep 09 20:00:11.033974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:00:11.033942    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:01:33.675898 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.675820    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc859cdcc-xh6sf"]
Sep 09 20:01:33.676322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.676135    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5801c08-63ca-4948-9d32-81a5fcb2195d" containerName="console"
Sep 09 20:01:33.676322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.676148    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5801c08-63ca-4948-9d32-81a5fcb2195d" containerName="console"
Sep 09 20:01:33.676322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.676217    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5801c08-63ca-4948-9d32-81a5fcb2195d" containerName="console"
Sep 09 20:01:33.679413 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.679392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.692433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.692409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc859cdcc-xh6sf"]
Sep 09 20:01:33.790498 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zjl\" (UniqueName: \"kubernetes.io/projected/211a4988-1056-48e7-8d0a-d277023a9110-kube-api-access-g9zjl\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-service-ca\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-trusted-ca-bundle\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-console-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-oauth-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.790702 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.790671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-oauth-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.891820 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-trusted-ca-bundle\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-console-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-oauth-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-oauth-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zjl\" (UniqueName: \"kubernetes.io/projected/211a4988-1056-48e7-8d0a-d277023a9110-kube-api-access-g9zjl\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892066 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.891986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-service-ca\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.892656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-service-ca\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.892656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-console-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.892744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-oauth-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.892818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.892807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211a4988-1056-48e7-8d0a-d277023a9110-trusted-ca-bundle\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.894832 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.894810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-oauth-config\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.894922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.894902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/211a4988-1056-48e7-8d0a-d277023a9110-console-serving-cert\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.901616 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.901594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zjl\" (UniqueName: \"kubernetes.io/projected/211a4988-1056-48e7-8d0a-d277023a9110-kube-api-access-g9zjl\") pod \"console-cc859cdcc-xh6sf\" (UID: \"211a4988-1056-48e7-8d0a-d277023a9110\") " pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:33.988773 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:33.988744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:34.264257 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:34.264233    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc859cdcc-xh6sf"]
Sep 09 20:01:34.266807 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:01:34.266783    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211a4988_1056_48e7_8d0a_d277023a9110.slice/crio-83c87d7147a7d4b42e972471cbf68a4faac7bbc1af8ff2b51baef06d53271838 WatchSource:0}: Error finding container 83c87d7147a7d4b42e972471cbf68a4faac7bbc1af8ff2b51baef06d53271838: Status 404 returned error can't find the container with id 83c87d7147a7d4b42e972471cbf68a4faac7bbc1af8ff2b51baef06d53271838
Sep 09 20:01:34.750393 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:34.750355    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc859cdcc-xh6sf" event={"ID":"211a4988-1056-48e7-8d0a-d277023a9110","Type":"ContainerStarted","Data":"7a37f500e2ff05f91ddca56ada1dd95067d6d0d8926bb54b3ace8997ec49e2c9"}
Sep 09 20:01:34.750393 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:34.750404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc859cdcc-xh6sf" event={"ID":"211a4988-1056-48e7-8d0a-d277023a9110","Type":"ContainerStarted","Data":"83c87d7147a7d4b42e972471cbf68a4faac7bbc1af8ff2b51baef06d53271838"}
Sep 09 20:01:34.770981 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:34.770901    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc859cdcc-xh6sf" podStartSLOduration=1.77088569 podStartE2EDuration="1.77088569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:34.769448471 +0000 UTC m=+384.170125674" watchObservedRunningTime="2026-09-09 20:01:34.77088569 +0000 UTC m=+384.171562882"
Sep 09 20:01:43.989454 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:43.989416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:43.989937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:43.989504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:43.994096 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:43.994073    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:44.792285 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:44.792252    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc859cdcc-xh6sf"
Sep 09 20:01:44.841493 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:44.841462    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 20:01:53.431425 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.431388    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"]
Sep 09 20:01:53.434791 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.434775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.440476 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.440450    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-djgrq\""
Sep 09 20:01:53.440593 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.440459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:01:53.440593 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.440569    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:01:53.440728 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.440660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:01:53.454969 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.454949    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"]
Sep 09 20:01:53.532000 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.531962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/21d052aa-44a6-408c-85df-a440b3b22b16-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.532000 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.532010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdqs\" (UniqueName: \"kubernetes.io/projected/21d052aa-44a6-408c-85df-a440b3b22b16-kube-api-access-4sdqs\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.632404 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.632369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/21d052aa-44a6-408c-85df-a440b3b22b16-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.632404 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.632403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdqs\" (UniqueName: \"kubernetes.io/projected/21d052aa-44a6-408c-85df-a440b3b22b16-kube-api-access-4sdqs\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.634626 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.634599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/21d052aa-44a6-408c-85df-a440b3b22b16-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.647783 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.647755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdqs\" (UniqueName: \"kubernetes.io/projected/21d052aa-44a6-408c-85df-a440b3b22b16-kube-api-access-4sdqs\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-7q7vw\" (UID: \"21d052aa-44a6-408c-85df-a440b3b22b16\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:53.744784 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:53.744755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:54.029893 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:54.029815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"]
Sep 09 20:01:54.032819 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:01:54.032789    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d052aa_44a6_408c_85df_a440b3b22b16.slice/crio-cc9031736ab2edc5f5e176f43efb20231311f79117a55a9150ed8e94a00ef56a WatchSource:0}: Error finding container cc9031736ab2edc5f5e176f43efb20231311f79117a55a9150ed8e94a00ef56a: Status 404 returned error can't find the container with id cc9031736ab2edc5f5e176f43efb20231311f79117a55a9150ed8e94a00ef56a
Sep 09 20:01:54.034851 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:54.034830    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:01:54.834615 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:54.834581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw" event={"ID":"21d052aa-44a6-408c-85df-a440b3b22b16","Type":"ContainerStarted","Data":"cc9031736ab2edc5f5e176f43efb20231311f79117a55a9150ed8e94a00ef56a"}
Sep 09 20:01:59.858148 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:59.858105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw" event={"ID":"21d052aa-44a6-408c-85df-a440b3b22b16","Type":"ContainerStarted","Data":"2f162b21e98018fa3d517031290fd3814591822657c47503be03143a912b7607"}
Sep 09 20:01:59.858531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:59.858283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:01:59.893258 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:01:59.893201    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw" podStartSLOduration=2.032902218 podStartE2EDuration="6.893186444s" totalImagesPullingTime="4.860284226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:59.890930183 +0000 UTC m=+409.291607376" watchObservedRunningTime="2026-09-09 20:01:59.893186444 +0000 UTC m=+409.293863637"
Sep 09 20:02:00.465448 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.465415    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-fg8qz"]
Sep 09 20:02:00.468807 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.468786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:00.471443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.471421    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:02:00.471575 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.471517    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:02:00.471575 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.471539    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-w8wgg\""
Sep 09 20:02:00.478612 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.478590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-fg8qz"]
Sep 09 20:02:00.490148 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.490125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btp9z\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-kube-api-access-btp9z\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:00.490238 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.490171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:00.591354 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.591322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:00.591502 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.591387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btp9z\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-kube-api-access-btp9z\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:00.591502 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:02:00.591475    2582 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 09 20:02:00.591502 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:02:00.591503    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-fg8qz: secret "keda-admission-webhooks-certs" not found
Sep 09 20:02:00.591602 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:02:00.591558    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates podName:c580498e-4f69-4c89-99a6-09aadf63a6f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:02:01.091537548 +0000 UTC m=+410.492214732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates") pod "keda-admission-6c4d864b86-fg8qz" (UID: "c580498e-4f69-4c89-99a6-09aadf63a6f1") : secret "keda-admission-webhooks-certs" not found
Sep 09 20:02:00.600855 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:00.600826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btp9z\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-kube-api-access-btp9z\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:01.095187 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:01.095155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:01.097616 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:01.097591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c580498e-4f69-4c89-99a6-09aadf63a6f1-certificates\") pod \"keda-admission-6c4d864b86-fg8qz\" (UID: \"c580498e-4f69-4c89-99a6-09aadf63a6f1\") " pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:01.379151 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:01.379067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:01.649605 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:01.649530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-fg8qz"]
Sep 09 20:02:01.652935 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:02:01.652900    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc580498e_4f69_4c89_99a6_09aadf63a6f1.slice/crio-cd45738690bff4fc89c81f95eb408bd0612afe4761d978750ddd5ec6e07df9b8 WatchSource:0}: Error finding container cd45738690bff4fc89c81f95eb408bd0612afe4761d978750ddd5ec6e07df9b8: Status 404 returned error can't find the container with id cd45738690bff4fc89c81f95eb408bd0612afe4761d978750ddd5ec6e07df9b8
Sep 09 20:02:01.866971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:01.866938    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz" event={"ID":"c580498e-4f69-4c89-99a6-09aadf63a6f1","Type":"ContainerStarted","Data":"cd45738690bff4fc89c81f95eb408bd0612afe4761d978750ddd5ec6e07df9b8"}
Sep 09 20:02:03.877184 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:03.877147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz" event={"ID":"c580498e-4f69-4c89-99a6-09aadf63a6f1","Type":"ContainerStarted","Data":"d46e0d68d024bf889dd2e20c9fc47657558c79532bf31b5b9fd820ddaa9e3283"}
Sep 09 20:02:03.877579 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:03.877207    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:02:03.895150 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:03.895102    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz" podStartSLOduration=2.307228821 podStartE2EDuration="3.895086747s" totalImagesPullingTime="1.587857926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:03.893822573 +0000 UTC m=+413.294499766" watchObservedRunningTime="2026-09-09 20:02:03.895086747 +0000 UTC m=+413.295763947"
Sep 09 20:02:09.861114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:09.861065    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-858d6458c8-bkk52" podUID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" containerName="console" containerID="cri-o://d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0" gracePeriod=15
Sep 09 20:02:10.105808 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.105784    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 20:02:10.167658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167632    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167878 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz27t\" (UniqueName: \"kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167878 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167710    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167878 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167743    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167878 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167777    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.167878 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.167796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config\") pod \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\" (UID: \"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f\") "
Sep 09 20:02:10.168171 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.168140    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:10.168242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.168141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:10.168383 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.168348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config" (OuterVolumeSpecName: "console-config") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:10.168511 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.168484    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:02:10.169817 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.169781    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:02:10.169913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.169828    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t" (OuterVolumeSpecName: "kube-api-access-kz27t") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "kube-api-access-kz27t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:02:10.169913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.169858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" (UID: "3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:02:10.268959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268923    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.268959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268951    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.268959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268960    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-trusted-ca-bundle\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.268959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268968    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-service-ca\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.269195 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268977    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz27t\" (UniqueName: \"kubernetes.io/projected/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-kube-api-access-kz27t\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.269195 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268986    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-console-oauth-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.269195 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.268994    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f-oauth-serving-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:02:10.909241 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.909203    2582 generic.go:356] "Generic (PLEG): container finished" podID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" containerID="d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0" exitCode=0
Sep 09 20:02:10.909712 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.909246    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858d6458c8-bkk52"
Sep 09 20:02:10.909712 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.909293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858d6458c8-bkk52" event={"ID":"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f","Type":"ContainerDied","Data":"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"}
Sep 09 20:02:10.909712 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.909349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858d6458c8-bkk52" event={"ID":"3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f","Type":"ContainerDied","Data":"f603998ae071b3dc8e0d38370fe6332132c3b32bf1b9fed1c76dbdeaf85f9bae"}
Sep 09 20:02:10.909712 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.909379    2582 scope.go:117] "RemoveContainer" containerID="d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"
Sep 09 20:02:10.917830 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.917812    2582 scope.go:117] "RemoveContainer" containerID="d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"
Sep 09 20:02:10.918093 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:02:10.918073    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0\": container with ID starting with d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0 not found: ID does not exist" containerID="d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"
Sep 09 20:02:10.918151 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.918107    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0"} err="failed to get container status \"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0\": rpc error: code = NotFound desc = could not find container \"d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0\": container with ID starting with d48451901063194f37496ba2f2646b76b03474ebe0230c36c91014565ce6cae0 not found: ID does not exist"
Sep 09 20:02:10.931002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.930980    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 20:02:10.935428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:10.935408    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858d6458c8-bkk52"]
Sep 09 20:02:11.123965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:11.123935    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" path="/var/lib/kubelet/pods/3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f/volumes"
Sep 09 20:02:20.863897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:20.863868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-7q7vw"
Sep 09 20:02:24.883807 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:02:24.883777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-fg8qz"
Sep 09 20:03:24.368508 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.368468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:03:24.369095 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.368985    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" containerName="console"
Sep 09 20:03:24.369095 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.369005    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" containerName="console"
Sep 09 20:03:24.369215 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.369097    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3804f434-d2a4-4fce-9e3c-5bb49b0ccf0f" containerName="console"
Sep 09 20:03:24.371010 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.370990    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.376990 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.376954    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:03:24.377139 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.376993    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:03:24.377139 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.377077    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-m7q29\""
Sep 09 20:03:24.377256 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.377222    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:03:24.396525 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.396499    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:03:24.423849 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.423814    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6kgl5"]
Sep 09 20:03:24.426070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.426053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.429888 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.429867    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:03:24.430169 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.430145    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cvxcb\""
Sep 09 20:03:24.437580 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.437557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6kgl5"]
Sep 09 20:03:24.449823 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.449800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq2ds\" (UniqueName: \"kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.449926 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.449875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.550471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.550436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.550658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.550509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq2ds\" (UniqueName: \"kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.550658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.550549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8eac914-d728-49e6-913b-ec115d861f8b-data\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.550658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.550577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmggv\" (UniqueName: \"kubernetes.io/projected/a8eac914-d728-49e6-913b-ec115d861f8b-kube-api-access-dmggv\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.552930 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.552898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.560490 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.560462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq2ds\" (UniqueName: \"kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds\") pod \"kserve-controller-manager-7676c798d9-6vm5v\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") " pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.651400 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.651321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8eac914-d728-49e6-913b-ec115d861f8b-data\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.651400 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.651356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmggv\" (UniqueName: \"kubernetes.io/projected/a8eac914-d728-49e6-913b-ec115d861f8b-kube-api-access-dmggv\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.651757 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.651738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8eac914-d728-49e6-913b-ec115d861f8b-data\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.662628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.662599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmggv\" (UniqueName: \"kubernetes.io/projected/a8eac914-d728-49e6-913b-ec115d861f8b-kube-api-access-dmggv\") pod \"seaweedfs-64568bcd49-6kgl5\" (UID: \"a8eac914-d728-49e6-913b-ec115d861f8b\") " pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.686510 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.686483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:24.735751 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.735719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:24.956965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:24.956941    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:03:24.959164 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:03:24.959136    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c616d_fd7a_42b2_af59_2b14b7288550.slice/crio-124bf5adbccb4ec5a91d4838c49a73c4b89979d8fd50291e6e28c9152aa236ef WatchSource:0}: Error finding container 124bf5adbccb4ec5a91d4838c49a73c4b89979d8fd50291e6e28c9152aa236ef: Status 404 returned error can't find the container with id 124bf5adbccb4ec5a91d4838c49a73c4b89979d8fd50291e6e28c9152aa236ef
Sep 09 20:03:25.015312 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:25.015231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6kgl5"]
Sep 09 20:03:25.017776 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:03:25.017740    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eac914_d728_49e6_913b_ec115d861f8b.slice/crio-4e95119b47dbc8c5fc13f83b7f29c3b172b9c7c53a358aa26472f60451786bde WatchSource:0}: Error finding container 4e95119b47dbc8c5fc13f83b7f29c3b172b9c7c53a358aa26472f60451786bde: Status 404 returned error can't find the container with id 4e95119b47dbc8c5fc13f83b7f29c3b172b9c7c53a358aa26472f60451786bde
Sep 09 20:03:25.212005 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:25.211917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" event={"ID":"261c616d-fd7a-42b2-af59-2b14b7288550","Type":"ContainerStarted","Data":"124bf5adbccb4ec5a91d4838c49a73c4b89979d8fd50291e6e28c9152aa236ef"}
Sep 09 20:03:25.213055 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:25.213031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6kgl5" event={"ID":"a8eac914-d728-49e6-913b-ec115d861f8b","Type":"ContainerStarted","Data":"4e95119b47dbc8c5fc13f83b7f29c3b172b9c7c53a358aa26472f60451786bde"}
Sep 09 20:03:29.234802 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:29.234699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" event={"ID":"261c616d-fd7a-42b2-af59-2b14b7288550","Type":"ContainerStarted","Data":"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"}
Sep 09 20:03:29.236590 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:29.236558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6kgl5" event={"ID":"a8eac914-d728-49e6-913b-ec115d861f8b","Type":"ContainerStarted","Data":"0a268f75047e8955d17fd192eb74ad1cf233541d8d4544c6341438b4c5a75acc"}
Sep 09 20:03:29.236749 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:29.236641    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:03:29.255075 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:29.255026    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" podStartSLOduration=1.113865761 podStartE2EDuration="5.255007922s" totalImagesPullingTime="4.141142161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:29.253560342 +0000 UTC m=+498.654237534" watchObservedRunningTime="2026-09-09 20:03:29.255007922 +0000 UTC m=+498.655685114"
Sep 09 20:03:29.274551 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:29.274504    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6kgl5" podStartSLOduration=1.160641703 podStartE2EDuration="5.274489415s" totalImagesPullingTime="4.113847712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:29.273785724 +0000 UTC m=+498.674462916" watchObservedRunningTime="2026-09-09 20:03:29.274489415 +0000 UTC m=+498.675166607"
Sep 09 20:03:30.240752 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:30.240720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:03:35.243034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:03:35.243004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6kgl5"
Sep 09 20:04:01.250325 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:01.250293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:04:09.042134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.042054    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:04:09.042601 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.042279    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" podUID="261c616d-fd7a-42b2-af59-2b14b7288550" containerName="manager" containerID="cri-o://93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01" gracePeriod=10
Sep 09 20:04:09.075541 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.075513    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-dqs57"]
Sep 09 20:04:09.078027 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.078012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.092588 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.092553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-dqs57"]
Sep 09 20:04:09.116955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.116935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98830e7-e434-41ec-81a1-0485483f2e82-cert\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.117053 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.116994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4d8\" (UniqueName: \"kubernetes.io/projected/d98830e7-e434-41ec-81a1-0485483f2e82-kube-api-access-qh4d8\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.218279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.218246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4d8\" (UniqueName: \"kubernetes.io/projected/d98830e7-e434-41ec-81a1-0485483f2e82-kube-api-access-qh4d8\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.218401 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.218329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98830e7-e434-41ec-81a1-0485483f2e82-cert\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.220845 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.220816    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98830e7-e434-41ec-81a1-0485483f2e82-cert\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.227832 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.227803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4d8\" (UniqueName: \"kubernetes.io/projected/d98830e7-e434-41ec-81a1-0485483f2e82-kube-api-access-qh4d8\") pod \"kserve-controller-manager-7676c798d9-dqs57\" (UID: \"d98830e7-e434-41ec-81a1-0485483f2e82\") " pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.275562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.275540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:04:09.318921 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.318859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq2ds\" (UniqueName: \"kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds\") pod \"261c616d-fd7a-42b2-af59-2b14b7288550\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") "
Sep 09 20:04:09.319043 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.318920    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert\") pod \"261c616d-fd7a-42b2-af59-2b14b7288550\" (UID: \"261c616d-fd7a-42b2-af59-2b14b7288550\") "
Sep 09 20:04:09.320994 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.320964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert" (OuterVolumeSpecName: "cert") pod "261c616d-fd7a-42b2-af59-2b14b7288550" (UID: "261c616d-fd7a-42b2-af59-2b14b7288550"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:04:09.321084 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.321051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds" (OuterVolumeSpecName: "kube-api-access-tq2ds") pod "261c616d-fd7a-42b2-af59-2b14b7288550" (UID: "261c616d-fd7a-42b2-af59-2b14b7288550"). InnerVolumeSpecName "kube-api-access-tq2ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:04:09.408553 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.408517    2582 generic.go:356] "Generic (PLEG): container finished" podID="261c616d-fd7a-42b2-af59-2b14b7288550" containerID="93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01" exitCode=0
Sep 09 20:04:09.408553 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.408558    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v"
Sep 09 20:04:09.408776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.408602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" event={"ID":"261c616d-fd7a-42b2-af59-2b14b7288550","Type":"ContainerDied","Data":"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"}
Sep 09 20:04:09.408776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.408651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-6vm5v" event={"ID":"261c616d-fd7a-42b2-af59-2b14b7288550","Type":"ContainerDied","Data":"124bf5adbccb4ec5a91d4838c49a73c4b89979d8fd50291e6e28c9152aa236ef"}
Sep 09 20:04:09.408776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.408673    2582 scope.go:117] "RemoveContainer" containerID="93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"
Sep 09 20:04:09.415747 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.415728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:09.417045 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.417027    2582 scope.go:117] "RemoveContainer" containerID="93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"
Sep 09 20:04:09.417395 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:04:09.417377    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01\": container with ID starting with 93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01 not found: ID does not exist" containerID="93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"
Sep 09 20:04:09.417474 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.417402    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01"} err="failed to get container status \"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01\": rpc error: code = NotFound desc = could not find container \"93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01\": container with ID starting with 93a421b58610cc66a36df37d687f79ee34d7a941874c2540555be2ac2ec80e01 not found: ID does not exist"
Sep 09 20:04:09.420153 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.420139    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c616d-fd7a-42b2-af59-2b14b7288550-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:04:09.420203 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.420155    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq2ds\" (UniqueName: \"kubernetes.io/projected/261c616d-fd7a-42b2-af59-2b14b7288550-kube-api-access-tq2ds\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:04:09.436277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.436256    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:04:09.439875 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.439854    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-6vm5v"]
Sep 09 20:04:09.684871 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:09.684661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-dqs57"]
Sep 09 20:04:09.687439 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:04:09.687415    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98830e7_e434_41ec_81a1_0485483f2e82.slice/crio-4e31722befeaa710abbf20b2b60df5d837ee752ac0cac8caf55e2b518cafee99 WatchSource:0}: Error finding container 4e31722befeaa710abbf20b2b60df5d837ee752ac0cac8caf55e2b518cafee99: Status 404 returned error can't find the container with id 4e31722befeaa710abbf20b2b60df5d837ee752ac0cac8caf55e2b518cafee99
Sep 09 20:04:10.414428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:10.414341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-dqs57" event={"ID":"d98830e7-e434-41ec-81a1-0485483f2e82","Type":"ContainerStarted","Data":"17c6918386e4b29ad821a01954d838272dbdcd6724919bf57cc0e31ec6f0ec1f"}
Sep 09 20:04:10.414428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:10.414390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-dqs57" event={"ID":"d98830e7-e434-41ec-81a1-0485483f2e82","Type":"ContainerStarted","Data":"4e31722befeaa710abbf20b2b60df5d837ee752ac0cac8caf55e2b518cafee99"}
Sep 09 20:04:10.414903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:10.414431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:10.432636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:10.432586    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-dqs57" podStartSLOduration=1.044538983 podStartE2EDuration="1.432571704s" totalImagesPullingTime="388.032721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:10.431604408 +0000 UTC m=+539.832281600" watchObservedRunningTime="2026-09-09 20:04:10.432571704 +0000 UTC m=+539.833248896"
Sep 09 20:04:11.127861 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:11.126201    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261c616d-fd7a-42b2-af59-2b14b7288550" path="/var/lib/kubelet/pods/261c616d-fd7a-42b2-af59-2b14b7288550/volumes"
Sep 09 20:04:41.424991 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:41.424959    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-dqs57"
Sep 09 20:04:42.905544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.905515    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2xc2s"]
Sep 09 20:04:42.905946 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.905935    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261c616d-fd7a-42b2-af59-2b14b7288550" containerName="manager"
Sep 09 20:04:42.905996 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.905949    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="261c616d-fd7a-42b2-af59-2b14b7288550" containerName="manager"
Sep 09 20:04:42.906033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.906011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="261c616d-fd7a-42b2-af59-2b14b7288550" containerName="manager"
Sep 09 20:04:42.908763 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.908747    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:42.911829 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.911812    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-grpps\""
Sep 09 20:04:42.912067 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.912049    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:04:42.917669 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.917645    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-dgbt7"]
Sep 09 20:04:42.920840 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.920823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:42.921359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.921339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2xc2s"]
Sep 09 20:04:42.922933 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.922902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:04:42.922933 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.922910    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:04:42.923065 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.922914    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-9lbxm\""
Sep 09 20:04:42.930233 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.930211    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-dgbt7"]
Sep 09 20:04:42.990512 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.990480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng758\" (UniqueName: \"kubernetes.io/projected/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-kube-api-access-ng758\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:42.990644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:42.990523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.090898 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.090865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-cert\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.091053 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.090918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7skx\" (UniqueName: \"kubernetes.io/projected/f6a322b8-ca57-416b-9c43-3464c1b2b357-kube-api-access-m7skx\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.091053 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.091005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng758\" (UniqueName: \"kubernetes.io/projected/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-kube-api-access-ng758\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.091167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.091052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.091167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.091075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.091276 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:04:43.091168    2582 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:04:43.091276 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:04:43.091243    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs podName:5325fcbe-4f01-4b6e-a7bd-7fedaf04a813 nodeName:}" failed. No retries permitted until 2026-09-09 20:04:43.591222349 +0000 UTC m=+572.991899533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs") pod "model-serving-api-7b954ffc95-2xc2s" (UID: "5325fcbe-4f01-4b6e-a7bd-7fedaf04a813") : secret "model-serving-api-tls" not found
Sep 09 20:04:43.099747 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.099726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng758\" (UniqueName: \"kubernetes.io/projected/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-kube-api-access-ng758\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.191556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.191483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7skx\" (UniqueName: \"kubernetes.io/projected/f6a322b8-ca57-416b-9c43-3464c1b2b357-kube-api-access-m7skx\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.191676 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.191579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.191676 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.191660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-cert\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.191778 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:04:43.191726    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 20:04:43.191813 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:04:43.191800    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs podName:f6a322b8-ca57-416b-9c43-3464c1b2b357 nodeName:}" failed. No retries permitted until 2026-09-09 20:04:43.691779803 +0000 UTC m=+573.092456985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs") pod "odh-model-controller-7b966d7984-dgbt7" (UID: "f6a322b8-ca57-416b-9c43-3464c1b2b357") : secret "odh-model-controller-metrics-tls" not found
Sep 09 20:04:43.199377 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.199352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-cert\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.201717 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.201675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7skx\" (UniqueName: \"kubernetes.io/projected/f6a322b8-ca57-416b-9c43-3464c1b2b357-kube-api-access-m7skx\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.595167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.595134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.597560 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.597524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5325fcbe-4f01-4b6e-a7bd-7fedaf04a813-tls-certs\") pod \"model-serving-api-7b954ffc95-2xc2s\" (UID: \"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813\") " pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.696415 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.696387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.698583 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.698565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a322b8-ca57-416b-9c43-3464c1b2b357-metrics-certs\") pod \"odh-model-controller-7b966d7984-dgbt7\" (UID: \"f6a322b8-ca57-416b-9c43-3464c1b2b357\") " pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:43.818992 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.818940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:43.831811 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:43.831786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:44.090816 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:44.090610    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2xc2s"]
Sep 09 20:04:44.092957 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:04:44.092933    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5325fcbe_4f01_4b6e_a7bd_7fedaf04a813.slice/crio-b45ae6c59c8322880a20e42522e72c01258015a12e9d92de39cab0ac0deb0744 WatchSource:0}: Error finding container b45ae6c59c8322880a20e42522e72c01258015a12e9d92de39cab0ac0deb0744: Status 404 returned error can't find the container with id b45ae6c59c8322880a20e42522e72c01258015a12e9d92de39cab0ac0deb0744
Sep 09 20:04:44.113182 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:44.113151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-dgbt7"]
Sep 09 20:04:44.115109 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:04:44.115056    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a322b8_ca57_416b_9c43_3464c1b2b357.slice/crio-b0605f4c33c7ac6e4c2fe9a98dd7b892ede488e1479b91a83d6606f19a975a43 WatchSource:0}: Error finding container b0605f4c33c7ac6e4c2fe9a98dd7b892ede488e1479b91a83d6606f19a975a43: Status 404 returned error can't find the container with id b0605f4c33c7ac6e4c2fe9a98dd7b892ede488e1479b91a83d6606f19a975a43
Sep 09 20:04:44.562268 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:44.562231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2xc2s" event={"ID":"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813","Type":"ContainerStarted","Data":"b45ae6c59c8322880a20e42522e72c01258015a12e9d92de39cab0ac0deb0744"}
Sep 09 20:04:44.563403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:44.563375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-dgbt7" event={"ID":"f6a322b8-ca57-416b-9c43-3464c1b2b357","Type":"ContainerStarted","Data":"b0605f4c33c7ac6e4c2fe9a98dd7b892ede488e1479b91a83d6606f19a975a43"}
Sep 09 20:04:46.574732 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:46.574633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2xc2s" event={"ID":"5325fcbe-4f01-4b6e-a7bd-7fedaf04a813","Type":"ContainerStarted","Data":"252d352308019eaacddc570a44c5d2980112fa73cc2bf7c7219aec238c2543f5"}
Sep 09 20:04:46.575153 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:46.574793    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:46.595758 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:46.595707    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2xc2s" podStartSLOduration=2.48054227 podStartE2EDuration="4.595692517s" totalImagesPullingTime="2.115150247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:46.593574295 +0000 UTC m=+575.994251488" watchObservedRunningTime="2026-09-09 20:04:46.595692517 +0000 UTC m=+575.996369699"
Sep 09 20:04:48.585817 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:48.585663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-dgbt7" event={"ID":"f6a322b8-ca57-416b-9c43-3464c1b2b357","Type":"ContainerStarted","Data":"55d4b9f2e1916bbd694eaad6d52ea8bb482b7c05615e149e365232d6c6f26ba1"}
Sep 09 20:04:48.585817 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:48.585747    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:04:57.583917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:57.583889    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2xc2s"
Sep 09 20:04:57.604216 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:57.604166    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-dgbt7" podStartSLOduration=11.504845607 podStartE2EDuration="15.604149523s" totalImagesPullingTime="4.099303916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:48.60774636 +0000 UTC m=+578.008423552" watchObservedRunningTime="2026-09-09 20:04:57.604149523 +0000 UTC m=+587.004826778"
Sep 09 20:04:59.592889 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:04:59.592861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-dgbt7"
Sep 09 20:06:22.755347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.755309    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:22.757739 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.757722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.760674 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.760649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 20:06:22.810813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.810779    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:22.830437 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.830411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjvjw\" (UniqueName: \"kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.830557 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.830445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.931225 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.931192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjvjw\" (UniqueName: \"kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.931343 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.931232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.931538 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.931523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:22.940899 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:22.940877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjvjw\" (UniqueName: \"kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw\") pod \"seaweedfs-tls-custom-6644bf5d6c-xn97r\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:23.067116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:23.067035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:23.340857 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:23.340828    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:23.343536 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:06:23.343508    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f82065a_1323_47ab_99cf_3edc5586dd81.slice/crio-00314c89f3005c09f627aec3f6774e7c2ba04e17444e3706289d8e819ea12857 WatchSource:0}: Error finding container 00314c89f3005c09f627aec3f6774e7c2ba04e17444e3706289d8e819ea12857: Status 404 returned error can't find the container with id 00314c89f3005c09f627aec3f6774e7c2ba04e17444e3706289d8e819ea12857
Sep 09 20:06:23.997639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:23.997556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" event={"ID":"4f82065a-1323-47ab-99cf-3edc5586dd81","Type":"ContainerStarted","Data":"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"}
Sep 09 20:06:23.997639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:23.997607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" event={"ID":"4f82065a-1323-47ab-99cf-3edc5586dd81","Type":"ContainerStarted","Data":"00314c89f3005c09f627aec3f6774e7c2ba04e17444e3706289d8e819ea12857"}
Sep 09 20:06:24.023855 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:24.023798    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" podStartSLOduration=1.643249665 podStartE2EDuration="2.023782994s" totalImagesPullingTime="380.533329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:24.023524023 +0000 UTC m=+673.424201388" watchObservedRunningTime="2026-09-09 20:06:24.023782994 +0000 UTC m=+673.424460185"
Sep 09 20:06:25.291312 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:25.291276    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:26.007808 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:26.007765    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" podUID="4f82065a-1323-47ab-99cf-3edc5586dd81" containerName="seaweedfs-tls-custom" containerID="cri-o://337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d" gracePeriod=30
Sep 09 20:06:27.248181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.248159    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:27.370867 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.370787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data\") pod \"4f82065a-1323-47ab-99cf-3edc5586dd81\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") "
Sep 09 20:06:27.370867 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.370828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjvjw\" (UniqueName: \"kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw\") pod \"4f82065a-1323-47ab-99cf-3edc5586dd81\" (UID: \"4f82065a-1323-47ab-99cf-3edc5586dd81\") "
Sep 09 20:06:27.372069 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.372038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data" (OuterVolumeSpecName: "data") pod "4f82065a-1323-47ab-99cf-3edc5586dd81" (UID: "4f82065a-1323-47ab-99cf-3edc5586dd81"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:06:27.372787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.372767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw" (OuterVolumeSpecName: "kube-api-access-gjvjw") pod "4f82065a-1323-47ab-99cf-3edc5586dd81" (UID: "4f82065a-1323-47ab-99cf-3edc5586dd81"). InnerVolumeSpecName "kube-api-access-gjvjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:06:27.471765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.471733    2582 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f82065a-1323-47ab-99cf-3edc5586dd81-data\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:06:27.471765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:27.471760    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjvjw\" (UniqueName: \"kubernetes.io/projected/4f82065a-1323-47ab-99cf-3edc5586dd81-kube-api-access-gjvjw\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:06:28.018173 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.018140    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f82065a-1323-47ab-99cf-3edc5586dd81" containerID="337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d" exitCode=0
Sep 09 20:06:28.018339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.018181    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"
Sep 09 20:06:28.018339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.018211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" event={"ID":"4f82065a-1323-47ab-99cf-3edc5586dd81","Type":"ContainerDied","Data":"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"}
Sep 09 20:06:28.018339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.018249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r" event={"ID":"4f82065a-1323-47ab-99cf-3edc5586dd81","Type":"ContainerDied","Data":"00314c89f3005c09f627aec3f6774e7c2ba04e17444e3706289d8e819ea12857"}
Sep 09 20:06:28.018339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.018268    2582 scope.go:117] "RemoveContainer" containerID="337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"
Sep 09 20:06:28.028949 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.028933    2582 scope.go:117] "RemoveContainer" containerID="337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"
Sep 09 20:06:28.029189 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:06:28.029169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d\": container with ID starting with 337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d not found: ID does not exist" containerID="337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"
Sep 09 20:06:28.029252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.029195    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d"} err="failed to get container status \"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d\": rpc error: code = NotFound desc = could not find container \"337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d\": container with ID starting with 337513be517cfa211dfb1da525b0dc40ee1aecc794136f589a1c21844407061d not found: ID does not exist"
Sep 09 20:06:28.042444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.042417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:28.046910 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.046890    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-xn97r"]
Sep 09 20:06:28.078993 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.078971    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"]
Sep 09 20:06:28.079311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.079298    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f82065a-1323-47ab-99cf-3edc5586dd81" containerName="seaweedfs-tls-custom"
Sep 09 20:06:28.079356 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.079313    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f82065a-1323-47ab-99cf-3edc5586dd81" containerName="seaweedfs-tls-custom"
Sep 09 20:06:28.079390 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.079381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f82065a-1323-47ab-99cf-3edc5586dd81" containerName="seaweedfs-tls-custom"
Sep 09 20:06:28.082161 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.082146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.084969 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.084948    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 20:06:28.085220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.085207    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 09 20:06:28.096276 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.096253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"]
Sep 09 20:06:28.177079 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.177052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.177230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.177089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kln\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-kube-api-access-76kln\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.177230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.177115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce703ffd-a1fc-4996-9694-7f648df0c9e5-data\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.278081 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.278011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.278081 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.278052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kln\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-kube-api-access-76kln\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.278081 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.278077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce703ffd-a1fc-4996-9694-7f648df0c9e5-data\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.278443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.278410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ce703ffd-a1fc-4996-9694-7f648df0c9e5-data\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.280569 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.280540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.288456 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.288426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kln\" (UniqueName: \"kubernetes.io/projected/ce703ffd-a1fc-4996-9694-7f648df0c9e5-kube-api-access-76kln\") pod \"seaweedfs-tls-custom-8b44d767d-xkg6j\" (UID: \"ce703ffd-a1fc-4996-9694-7f648df0c9e5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.390900 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.390866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"
Sep 09 20:06:28.661903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:28.661828    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-xkg6j"]
Sep 09 20:06:28.665311 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:06:28.665285    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce703ffd_a1fc_4996_9694_7f648df0c9e5.slice/crio-aa974d9c116d6521a11aa897e93a0c55d4f7e557088f6244d035270a721650d7 WatchSource:0}: Error finding container aa974d9c116d6521a11aa897e93a0c55d4f7e557088f6244d035270a721650d7: Status 404 returned error can't find the container with id aa974d9c116d6521a11aa897e93a0c55d4f7e557088f6244d035270a721650d7
Sep 09 20:06:29.025964 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:29.025927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j" event={"ID":"ce703ffd-a1fc-4996-9694-7f648df0c9e5","Type":"ContainerStarted","Data":"0b4e16fd9b27aa375e4cb258f6ca99bfdbf1ddb871a5eb552a683eb7a1eccb71"}
Sep 09 20:06:29.026106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:29.025989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j" event={"ID":"ce703ffd-a1fc-4996-9694-7f648df0c9e5","Type":"ContainerStarted","Data":"aa974d9c116d6521a11aa897e93a0c55d4f7e557088f6244d035270a721650d7"}
Sep 09 20:06:29.045230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:29.045187    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-xkg6j" podStartSLOduration=0.778010696 podStartE2EDuration="1.045172484s" totalImagesPullingTime="267.161788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:29.043029857 +0000 UTC m=+678.443707060" watchObservedRunningTime="2026-09-09 20:06:29.045172484 +0000 UTC m=+678.445849676"
Sep 09 20:06:29.125007 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:06:29.124976    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f82065a-1323-47ab-99cf-3edc5586dd81" path="/var/lib/kubelet/pods/4f82065a-1323-47ab-99cf-3edc5586dd81/volumes"
Sep 09 20:07:13.720307 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.720268    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:13.723241 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.723224    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.727179 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727139    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-705883-predictor-serving-cert\""
Sep 09 20:07:13.727370 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727351    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:07:13.727444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727395    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-705883\""
Sep 09 20:07:13.727947 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:07:13.728059 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:07:13.728059 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.727958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-705883-kube-rbac-proxy-sar-config\""
Sep 09 20:07:13.728240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.728225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-705883-dockercfg-kskqb\""
Sep 09 20:07:13.737200 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.737179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:13.850716 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.850664    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.850892 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.850730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.850892 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.850811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.850892 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.850860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.851046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.850938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg52q\" (UniqueName: \"kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.951578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.951543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.951750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.951589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.951750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.951645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg52q\" (UniqueName: \"kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.951750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.951722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.951897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.951761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.952144 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.952123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.952405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.952382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.952481 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.952382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.954193 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.954170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:13.961806 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:13.961778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg52q\" (UniqueName: \"kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q\") pod \"isvc-init-fail-705883-predictor-679b595fd5-bk5xb\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") " pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:14.033614 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:14.033555    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:14.311861 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:14.311829    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:14.313604 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:07:14.313578    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4158f04e_6b7d_451f_bb47_14500f88ee6b.slice/crio-a633e338f9c5a7bfcdea1b0ebb0feb505ad67b85aa7ef905bf5106b5eca6c2fa WatchSource:0}: Error finding container a633e338f9c5a7bfcdea1b0ebb0feb505ad67b85aa7ef905bf5106b5eca6c2fa: Status 404 returned error can't find the container with id a633e338f9c5a7bfcdea1b0ebb0feb505ad67b85aa7ef905bf5106b5eca6c2fa
Sep 09 20:07:14.316655 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:14.316637    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:07:15.231252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:15.231214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerStarted","Data":"a633e338f9c5a7bfcdea1b0ebb0feb505ad67b85aa7ef905bf5106b5eca6c2fa"}
Sep 09 20:07:18.249557 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:18.249519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerStarted","Data":"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"}
Sep 09 20:07:21.263366 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:21.263335    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/0.log"
Sep 09 20:07:21.263839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:21.263375    2582 generic.go:356] "Generic (PLEG): container finished" podID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerID="6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141" exitCode=1
Sep 09 20:07:21.263839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:21.263413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerDied","Data":"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"}
Sep 09 20:07:22.269406 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:22.269378    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/0.log"
Sep 09 20:07:22.269774 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:22.269436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerStarted","Data":"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"}
Sep 09 20:07:23.340860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:23.340826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:23.341214 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:23.341012    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer" containerID="cri-o://0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23" gracePeriod=30
Sep 09 20:07:27.983943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:27.983920    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/1.log"
Sep 09 20:07:27.984350 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:27.984335    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/0.log"
Sep 09 20:07:27.985594 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:27.985579    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:28.072641 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg52q\" (UniqueName: \"kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q\") pod \"4158f04e-6b7d-451f-bb47-14500f88ee6b\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") "
Sep 09 20:07:28.072641 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location\") pod \"4158f04e-6b7d-451f-bb47-14500f88ee6b\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") "
Sep 09 20:07:28.072641 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config\") pod \"4158f04e-6b7d-451f-bb47-14500f88ee6b\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") "
Sep 09 20:07:28.072882 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls\") pod \"4158f04e-6b7d-451f-bb47-14500f88ee6b\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") "
Sep 09 20:07:28.072882 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072792    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert\") pod \"4158f04e-6b7d-451f-bb47-14500f88ee6b\" (UID: \"4158f04e-6b7d-451f-bb47-14500f88ee6b\") "
Sep 09 20:07:28.072970 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.072923    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4158f04e-6b7d-451f-bb47-14500f88ee6b" (UID: "4158f04e-6b7d-451f-bb47-14500f88ee6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:07:28.073023 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.073003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4158f04e-6b7d-451f-bb47-14500f88ee6b" (UID: "4158f04e-6b7d-451f-bb47-14500f88ee6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:07:28.073173 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.073154    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4158f04e-6b7d-451f-bb47-14500f88ee6b-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:07:28.073259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.073177    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:07:28.073259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.073196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4158f04e-6b7d-451f-bb47-14500f88ee6b" (UID: "4158f04e-6b7d-451f-bb47-14500f88ee6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:07:28.074829 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.074800    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4158f04e-6b7d-451f-bb47-14500f88ee6b" (UID: "4158f04e-6b7d-451f-bb47-14500f88ee6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:07:28.074933 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.074864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q" (OuterVolumeSpecName: "kube-api-access-zg52q") pod "4158f04e-6b7d-451f-bb47-14500f88ee6b" (UID: "4158f04e-6b7d-451f-bb47-14500f88ee6b"). InnerVolumeSpecName "kube-api-access-zg52q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:07:28.174573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.174539    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4158f04e-6b7d-451f-bb47-14500f88ee6b-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:07:28.174573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.174567    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4158f04e-6b7d-451f-bb47-14500f88ee6b-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:07:28.174573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.174578    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg52q\" (UniqueName: \"kubernetes.io/projected/4158f04e-6b7d-451f-bb47-14500f88ee6b-kube-api-access-zg52q\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:07:28.298796 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.298771    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/1.log"
Sep 09 20:07:28.299180 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299164    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-705883-predictor-679b595fd5-bk5xb_4158f04e-6b7d-451f-bb47-14500f88ee6b/storage-initializer/0.log"
Sep 09 20:07:28.299277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299205    2582 generic.go:356] "Generic (PLEG): container finished" podID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerID="0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23" exitCode=1
Sep 09 20:07:28.299277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerDied","Data":"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"}
Sep 09 20:07:28.299277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299254    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"
Sep 09 20:07:28.299428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb" event={"ID":"4158f04e-6b7d-451f-bb47-14500f88ee6b","Type":"ContainerDied","Data":"a633e338f9c5a7bfcdea1b0ebb0feb505ad67b85aa7ef905bf5106b5eca6c2fa"}
Sep 09 20:07:28.299428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.299321    2582 scope.go:117] "RemoveContainer" containerID="0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"
Sep 09 20:07:28.307310 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.307258    2582 scope.go:117] "RemoveContainer" containerID="6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"
Sep 09 20:07:28.314539 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.314521    2582 scope.go:117] "RemoveContainer" containerID="0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"
Sep 09 20:07:28.314794 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:07:28.314772    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23\": container with ID starting with 0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23 not found: ID does not exist" containerID="0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"
Sep 09 20:07:28.314864 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.314802    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23"} err="failed to get container status \"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23\": rpc error: code = NotFound desc = could not find container \"0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23\": container with ID starting with 0c1d00cc60bb4031434a722ef3feb64138b7f947b984ccb36c808b62a83d6e23 not found: ID does not exist"
Sep 09 20:07:28.314864 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.314817    2582 scope.go:117] "RemoveContainer" containerID="6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"
Sep 09 20:07:28.315053 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:07:28.315040    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141\": container with ID starting with 6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141 not found: ID does not exist" containerID="6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"
Sep 09 20:07:28.315106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.315056    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141"} err="failed to get container status \"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141\": rpc error: code = NotFound desc = could not find container \"6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141\": container with ID starting with 6d5017de500ce11a9a733ae87ec76ebb6ddda8ce4b00c8ddec159a492d46f141 not found: ID does not exist"
Sep 09 20:07:28.341788 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.341728    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:28.345709 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.345672    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-705883-predictor-679b595fd5-bk5xb"]
Sep 09 20:07:28.613793 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.613717    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:07:28.614094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614081    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.614136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.614136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614110    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.614136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614116    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.614228 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614176    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.614228 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.614184    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" containerName="storage-initializer"
Sep 09 20:07:28.617381 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.617365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.620508 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620486    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 09 20:07:28.620639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620519    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:07:28.620639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620599    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:07:28.620786 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620620    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:07:28.620786 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620739    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:07:28.620900 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.620817    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 20:07:28.621217 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.621204    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:07:28.631854 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.631835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:07:28.780825 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.780790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.781029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.780841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.781029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.780900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf8zr\" (UniqueName: \"kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.781029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.780953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.781029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.780994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.881932 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.881846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.881932 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.881910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882142 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.881944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf8zr\" (UniqueName: \"kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882142 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.881969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882260 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.882162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882563 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.882543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882690 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.882657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.882738 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.882660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.884792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.884772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.893403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.893374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf8zr\" (UniqueName: \"kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-cspf7\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:28.927256 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:28.927229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:07:29.124929 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:29.124896    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4158f04e-6b7d-451f-bb47-14500f88ee6b" path="/var/lib/kubelet/pods/4158f04e-6b7d-451f-bb47-14500f88ee6b/volumes"
Sep 09 20:07:29.198382 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:29.198356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:07:29.200832 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:07:29.200806    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27acf36_e094_4e2b_94b8_29a8b7b7af14.slice/crio-c884b88e27885c35610df37658d7ae6a7d220611c188665e62548b5604663281 WatchSource:0}: Error finding container c884b88e27885c35610df37658d7ae6a7d220611c188665e62548b5604663281: Status 404 returned error can't find the container with id c884b88e27885c35610df37658d7ae6a7d220611c188665e62548b5604663281
Sep 09 20:07:29.307283 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:29.307246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerStarted","Data":"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"}
Sep 09 20:07:29.307422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:29.307321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerStarted","Data":"c884b88e27885c35610df37658d7ae6a7d220611c188665e62548b5604663281"}
Sep 09 20:07:33.325728 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:33.325696    2582 generic.go:356] "Generic (PLEG): container finished" podID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerID="5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481" exitCode=0
Sep 09 20:07:33.326045 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:33.325756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerDied","Data":"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"}
Sep 09 20:07:58.476965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:07:58.476928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerStarted","Data":"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"}
Sep 09 20:08:01.492879 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:01.492845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerStarted","Data":"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"}
Sep 09 20:08:01.493242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:01.492989    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:08:01.516315 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:01.516271    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podStartSLOduration=2.644649598 podStartE2EDuration="33.516251616s" totalImagesPullingTime="26.917546181s" totalInitContainerRuntime="3.954055837s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:07:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:01.513460657 +0000 UTC m=+770.914137850" watchObservedRunningTime="2026-09-09 20:08:01.516251616 +0000 UTC m=+770.916928808"
Sep 09 20:08:02.497472 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:02.497435    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:08:02.498552 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:02.498526    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:03.502060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:03.502022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:08.506587 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:08.506556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:08:08.507060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:08.507034    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:18.507525 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:18.507479    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:28.507005 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:28.506961    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:38.507578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:38.507493    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:48.507992 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:48.507950    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:08:58.507177 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:08:58.507138    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:09:08.507114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:08.507073    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 09 20:09:18.508331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:18.508302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:09:28.661742 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:28.661709    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:09:28.662146 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:28.661969    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container" containerID="cri-o://43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe" gracePeriod=30
Sep 09 20:09:28.662146 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:28.662007    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kube-rbac-proxy" containerID="cri-o://b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5" gracePeriod=30
Sep 09 20:09:28.912641 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:28.912561    2582 generic.go:356] "Generic (PLEG): container finished" podID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerID="b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5" exitCode=2
Sep 09 20:09:28.912820 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:28.912632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerDied","Data":"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"}
Sep 09 20:09:33.097971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.097947    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:09:33.144826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.144761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location\") pod \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") "
Sep 09 20:09:33.144826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.144796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config\") pod \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") "
Sep 09 20:09:33.144826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.144826    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf8zr\" (UniqueName: \"kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr\") pod \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") "
Sep 09 20:09:33.145085 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.144850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls\") pod \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") "
Sep 09 20:09:33.145085 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.144877    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert\") pod \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\" (UID: \"b27acf36-e094-4e2b-94b8-29a8b7b7af14\") "
Sep 09 20:09:33.145208 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.145179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b27acf36-e094-4e2b-94b8-29a8b7b7af14" (UID: "b27acf36-e094-4e2b-94b8-29a8b7b7af14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:33.145254 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.145118    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b27acf36-e094-4e2b-94b8-29a8b7b7af14" (UID: "b27acf36-e094-4e2b-94b8-29a8b7b7af14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:33.145289 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.145273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b27acf36-e094-4e2b-94b8-29a8b7b7af14" (UID: "b27acf36-e094-4e2b-94b8-29a8b7b7af14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:33.146942 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.146914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr" (OuterVolumeSpecName: "kube-api-access-nf8zr") pod "b27acf36-e094-4e2b-94b8-29a8b7b7af14" (UID: "b27acf36-e094-4e2b-94b8-29a8b7b7af14"). InnerVolumeSpecName "kube-api-access-nf8zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:09:33.147025 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.146984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b27acf36-e094-4e2b-94b8-29a8b7b7af14" (UID: "b27acf36-e094-4e2b-94b8-29a8b7b7af14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:09:33.246282 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.246244    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:09:33.246282 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.246279    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:09:33.246434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.246293    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf8zr\" (UniqueName: \"kubernetes.io/projected/b27acf36-e094-4e2b-94b8-29a8b7b7af14-kube-api-access-nf8zr\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:09:33.246434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.246307    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b27acf36-e094-4e2b-94b8-29a8b7b7af14-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:09:33.246434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.246322    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b27acf36-e094-4e2b-94b8-29a8b7b7af14-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:09:33.854315 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854282    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:09:33.854757 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854739    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kube-rbac-proxy"
Sep 09 20:09:33.854872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854761    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kube-rbac-proxy"
Sep 09 20:09:33.854872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854784    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="storage-initializer"
Sep 09 20:09:33.854872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854792    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="storage-initializer"
Sep 09 20:09:33.854872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854808    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container"
Sep 09 20:09:33.854872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854816    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container"
Sep 09 20:09:33.855122 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854922    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kserve-container"
Sep 09 20:09:33.855122 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.854938    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerName="kube-rbac-proxy"
Sep 09 20:09:33.857313 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.857295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.859559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.859536    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 20:09:33.859836 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.859821    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 09 20:09:33.866974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.866953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:09:33.936069 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.936035    2582 generic.go:356] "Generic (PLEG): container finished" podID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" containerID="43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe" exitCode=0
Sep 09 20:09:33.936220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.936102    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"
Sep 09 20:09:33.936220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.936120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerDied","Data":"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"}
Sep 09 20:09:33.936220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.936194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7" event={"ID":"b27acf36-e094-4e2b-94b8-29a8b7b7af14","Type":"ContainerDied","Data":"c884b88e27885c35610df37658d7ae6a7d220611c188665e62548b5604663281"}
Sep 09 20:09:33.936379 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.936232    2582 scope.go:117] "RemoveContainer" containerID="b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"
Sep 09 20:09:33.944884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.944863    2582 scope.go:117] "RemoveContainer" containerID="43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"
Sep 09 20:09:33.952394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952372    2582 scope.go:117] "RemoveContainer" containerID="5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"
Sep 09 20:09:33.952544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.952666 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.952666 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.952797 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.952797 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.952750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67k9\" (UniqueName: \"kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:33.958604 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.958582    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:09:33.959608 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.959593    2582 scope.go:117] "RemoveContainer" containerID="b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"
Sep 09 20:09:33.959875 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:09:33.959859    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5\": container with ID starting with b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5 not found: ID does not exist" containerID="b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"
Sep 09 20:09:33.959937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.959880    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5"} err="failed to get container status \"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5\": rpc error: code = NotFound desc = could not find container \"b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5\": container with ID starting with b41bef52af7b5a7d64ba5411bbe7253baaee7752d1b164c853c3010db879b5a5 not found: ID does not exist"
Sep 09 20:09:33.959937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.959895    2582 scope.go:117] "RemoveContainer" containerID="43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"
Sep 09 20:09:33.960138 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:09:33.960119    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe\": container with ID starting with 43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe not found: ID does not exist" containerID="43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"
Sep 09 20:09:33.960181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.960145    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe"} err="failed to get container status \"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe\": rpc error: code = NotFound desc = could not find container \"43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe\": container with ID starting with 43642302413d498db40d03367f1cf82a22e16c20ec746498ad3a78d1a1872bfe not found: ID does not exist"
Sep 09 20:09:33.960181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.960165    2582 scope.go:117] "RemoveContainer" containerID="5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"
Sep 09 20:09:33.960426 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:09:33.960407    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481\": container with ID starting with 5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481 not found: ID does not exist" containerID="5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"
Sep 09 20:09:33.960481 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.960430    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481"} err="failed to get container status \"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481\": rpc error: code = NotFound desc = could not find container \"5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481\": container with ID starting with 5ac6d65585217fe30864c2801526a789f33fb0b6ec053e1075099a9866844481 not found: ID does not exist"
Sep 09 20:09:33.964644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:33.964624    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-cspf7"]
Sep 09 20:09:34.053751 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.053726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.053852 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.053758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.053852 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.053792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67k9\" (UniqueName: \"kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.053936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.053858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.053936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.053904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.054243 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.054223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.054415 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.054396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.054466 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.054445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.056088 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.056071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.062327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.062306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67k9\" (UniqueName: \"kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9\") pod \"isvc-predictive-xgboost-predictor-695f978978-5w7tv\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.168511 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.168439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:34.438457 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.438430    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:09:34.441097 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:09:34.441064    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ffffce_98c8_4a8b_8189_14960c366080.slice/crio-31e7e96ca5b6e980a3003149fcc7ffe7a261282207dbc6b84cb69f19acd4c1d6 WatchSource:0}: Error finding container 31e7e96ca5b6e980a3003149fcc7ffe7a261282207dbc6b84cb69f19acd4c1d6: Status 404 returned error can't find the container with id 31e7e96ca5b6e980a3003149fcc7ffe7a261282207dbc6b84cb69f19acd4c1d6
Sep 09 20:09:34.942194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.942154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerStarted","Data":"d82582ad640e676c6a601034834c9e55477aeeca63029894b869940d161fe0f4"}
Sep 09 20:09:34.942365 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:34.942219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerStarted","Data":"31e7e96ca5b6e980a3003149fcc7ffe7a261282207dbc6b84cb69f19acd4c1d6"}
Sep 09 20:09:35.124877 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:35.124842    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27acf36-e094-4e2b-94b8-29a8b7b7af14" path="/var/lib/kubelet/pods/b27acf36-e094-4e2b-94b8-29a8b7b7af14/volumes"
Sep 09 20:09:38.962292 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:38.962254    2582 generic.go:356] "Generic (PLEG): container finished" podID="94ffffce-98c8-4a8b-8189-14960c366080" containerID="d82582ad640e676c6a601034834c9e55477aeeca63029894b869940d161fe0f4" exitCode=0
Sep 09 20:09:38.962787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:38.962336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerDied","Data":"d82582ad640e676c6a601034834c9e55477aeeca63029894b869940d161fe0f4"}
Sep 09 20:09:39.968640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.968605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerStarted","Data":"767d15458ad8996bc1f1cd081fbf4318b73c16a0d5823e655b9af293776c9933"}
Sep 09 20:09:39.969152 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.968668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerStarted","Data":"5bef1b4513174aa36f46b0cf37f04494b988033065336b0efa8bc7881128459d"}
Sep 09 20:09:39.969152 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.968960    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:39.969152 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.969109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:39.970371 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.970348    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:09:39.991307 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:39.991258    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podStartSLOduration=3.110585648 podStartE2EDuration="6.991244203s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880658555s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:09:39.988910185 +0000 UTC m=+869.389587381" watchObservedRunningTime="2026-09-09 20:09:39.991244203 +0000 UTC m=+869.391921396"
Sep 09 20:09:40.973546 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:40.973506    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:09:45.978354 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:45.978324    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:09:45.978836 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:45.978811    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:09:55.978841 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:09:55.978798    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:05.979266 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:05.979152    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:15.979043 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:15.979005    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:23.674696 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.674654    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:10:23.679350 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.679329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.681987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.681956    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 09 20:10:23.681987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.681964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 09 20:10:23.687077 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.687055    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:10:23.784630 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.784591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb75z\" (UniqueName: \"kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.784813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.784704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.784813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.784755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.885441 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.885403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb75z\" (UniqueName: \"kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.885626 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.885464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.885626 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.885489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.886171 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.886148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.888048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.888020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.901617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.901588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb75z\" (UniqueName: \"kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z\") pod \"message-dumper-predictor-6575bcfdfd-82b2z\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:23.990266 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:23.990241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:24.263013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:24.262986    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:10:24.265812 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:10:24.265786    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd1c18_7540_414e_bece_7d03fadcb8d4.slice/crio-11d5302e195da1e2647819b9ce9c68302242e94341fb7703f40cd5d478a0133f WatchSource:0}: Error finding container 11d5302e195da1e2647819b9ce9c68302242e94341fb7703f40cd5d478a0133f: Status 404 returned error can't find the container with id 11d5302e195da1e2647819b9ce9c68302242e94341fb7703f40cd5d478a0133f
Sep 09 20:10:25.173966 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:25.173930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerStarted","Data":"11d5302e195da1e2647819b9ce9c68302242e94341fb7703f40cd5d478a0133f"}
Sep 09 20:10:25.978807 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:25.978772    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:26.180598 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.180555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerStarted","Data":"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"}
Sep 09 20:10:26.181042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.180627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerStarted","Data":"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"}
Sep 09 20:10:26.181042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.180769    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:26.181042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.180801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:26.182526 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.182503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:26.202892 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:26.202843    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" podStartSLOduration=1.968141294 podStartE2EDuration="3.202829831s" totalImagesPullingTime="1.234688537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:26.201427771 +0000 UTC m=+915.602104964" watchObservedRunningTime="2026-09-09 20:10:26.202829831 +0000 UTC m=+915.603507061"
Sep 09 20:10:33.194997 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:33.194969    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:10:35.979098 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:35.979055    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:45.978814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:45.978773    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 09 20:10:51.125069 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:51.125037    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:10:53.936013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:53.935983    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:10:53.936420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:53.936184    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container" containerID="cri-o://5bef1b4513174aa36f46b0cf37f04494b988033065336b0efa8bc7881128459d" gracePeriod=30
Sep 09 20:10:53.936420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:53.936237    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kube-rbac-proxy" containerID="cri-o://767d15458ad8996bc1f1cd081fbf4318b73c16a0d5823e655b9af293776c9933" gracePeriod=30
Sep 09 20:10:54.311498 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:54.311463    2582 generic.go:356] "Generic (PLEG): container finished" podID="94ffffce-98c8-4a8b-8189-14960c366080" containerID="767d15458ad8996bc1f1cd081fbf4318b73c16a0d5823e655b9af293776c9933" exitCode=2
Sep 09 20:10:54.311656 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:54.311534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerDied","Data":"767d15458ad8996bc1f1cd081fbf4318b73c16a0d5823e655b9af293776c9933"}
Sep 09 20:10:55.973942 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:55.973896    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 09 20:10:58.333450 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.333417    2582 generic.go:356] "Generic (PLEG): container finished" podID="94ffffce-98c8-4a8b-8189-14960c366080" containerID="5bef1b4513174aa36f46b0cf37f04494b988033065336b0efa8bc7881128459d" exitCode=0
Sep 09 20:10:58.333785 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.333460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerDied","Data":"5bef1b4513174aa36f46b0cf37f04494b988033065336b0efa8bc7881128459d"}
Sep 09 20:10:58.383065 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.383043    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:10:58.476867 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.476838    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config\") pod \"94ffffce-98c8-4a8b-8189-14960c366080\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") "
Sep 09 20:10:58.477006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.476895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert\") pod \"94ffffce-98c8-4a8b-8189-14960c366080\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") "
Sep 09 20:10:58.477006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.476958    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls\") pod \"94ffffce-98c8-4a8b-8189-14960c366080\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") "
Sep 09 20:10:58.477006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.476981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location\") pod \"94ffffce-98c8-4a8b-8189-14960c366080\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") "
Sep 09 20:10:58.477156 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.477028    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t67k9\" (UniqueName: \"kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9\") pod \"94ffffce-98c8-4a8b-8189-14960c366080\" (UID: \"94ffffce-98c8-4a8b-8189-14960c366080\") "
Sep 09 20:10:58.477303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.477274    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94ffffce-98c8-4a8b-8189-14960c366080" (UID: "94ffffce-98c8-4a8b-8189-14960c366080"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:58.477409 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.477317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94ffffce-98c8-4a8b-8189-14960c366080" (UID: "94ffffce-98c8-4a8b-8189-14960c366080"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:58.477409 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.477346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94ffffce-98c8-4a8b-8189-14960c366080" (UID: "94ffffce-98c8-4a8b-8189-14960c366080"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:58.478976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.478955    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94ffffce-98c8-4a8b-8189-14960c366080" (UID: "94ffffce-98c8-4a8b-8189-14960c366080"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:58.479100 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.479084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9" (OuterVolumeSpecName: "kube-api-access-t67k9") pod "94ffffce-98c8-4a8b-8189-14960c366080" (UID: "94ffffce-98c8-4a8b-8189-14960c366080"). InnerVolumeSpecName "kube-api-access-t67k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:58.578519 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.578484    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67k9\" (UniqueName: \"kubernetes.io/projected/94ffffce-98c8-4a8b-8189-14960c366080-kube-api-access-t67k9\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:10:58.578519 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.578514    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:10:58.578519 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.578523    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ffffce-98c8-4a8b-8189-14960c366080-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:10:58.578784 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.578532    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ffffce-98c8-4a8b-8189-14960c366080-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:10:58.578784 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:58.578541    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ffffce-98c8-4a8b-8189-14960c366080-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:10:59.196087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196054    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:10:59.196423 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196410    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="storage-initializer"
Sep 09 20:10:59.196471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196425    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="storage-initializer"
Sep 09 20:10:59.196471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196445    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kube-rbac-proxy"
Sep 09 20:10:59.196471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196450    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kube-rbac-proxy"
Sep 09 20:10:59.196471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container"
Sep 09 20:10:59.196471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196469    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container"
Sep 09 20:10:59.196636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196520    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kube-rbac-proxy"
Sep 09 20:10:59.196636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.196532    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ffffce-98c8-4a8b-8189-14960c366080" containerName="kserve-container"
Sep 09 20:10:59.199907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.199876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.202568 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.202544    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 09 20:10:59.202735 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.202718    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 20:10:59.214210 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.214191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:10:59.284866 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.284833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjltv\" (UniqueName: \"kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.285041 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.284871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.285041 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.284891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.285041 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.284998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.285197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.285100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.340932 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.340907    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"
Sep 09 20:10:59.341302 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.340926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv" event={"ID":"94ffffce-98c8-4a8b-8189-14960c366080","Type":"ContainerDied","Data":"31e7e96ca5b6e980a3003149fcc7ffe7a261282207dbc6b84cb69f19acd4c1d6"}
Sep 09 20:10:59.341302 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.340994    2582 scope.go:117] "RemoveContainer" containerID="767d15458ad8996bc1f1cd081fbf4318b73c16a0d5823e655b9af293776c9933"
Sep 09 20:10:59.349942 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.349921    2582 scope.go:117] "RemoveContainer" containerID="5bef1b4513174aa36f46b0cf37f04494b988033065336b0efa8bc7881128459d"
Sep 09 20:10:59.357389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.357374    2582 scope.go:117] "RemoveContainer" containerID="d82582ad640e676c6a601034834c9e55477aeeca63029894b869940d161fe0f4"
Sep 09 20:10:59.362443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.362421    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:10:59.365532 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.365506    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5w7tv"]
Sep 09 20:10:59.386362 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386362 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386477 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386477 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjltv\" (UniqueName: \"kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386477 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386812 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386792    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.386959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.386942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.387025 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.387009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.388737 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.388720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.395545 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.395522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjltv\" (UniqueName: \"kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.509848 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.509816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:10:59.787194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:10:59.787081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:10:59.789536 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:10:59.789510    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ff8ecf_68c9_4266_9b52_ccbec3607cca.slice/crio-3f5838f1340ab9430a9490b6bc43245a4d2ad8ef1891a63fec9ab345e2cb6a8b WatchSource:0}: Error finding container 3f5838f1340ab9430a9490b6bc43245a4d2ad8ef1891a63fec9ab345e2cb6a8b: Status 404 returned error can't find the container with id 3f5838f1340ab9430a9490b6bc43245a4d2ad8ef1891a63fec9ab345e2cb6a8b
Sep 09 20:11:00.347810 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:00.347771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerStarted","Data":"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"}
Sep 09 20:11:00.348169 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:00.347830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerStarted","Data":"3f5838f1340ab9430a9490b6bc43245a4d2ad8ef1891a63fec9ab345e2cb6a8b"}
Sep 09 20:11:01.124105 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:01.124067    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ffffce-98c8-4a8b-8189-14960c366080" path="/var/lib/kubelet/pods/94ffffce-98c8-4a8b-8189-14960c366080/volumes"
Sep 09 20:11:04.366485 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:04.366451    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerID="3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010" exitCode=0
Sep 09 20:11:04.366983 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:04.366503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerDied","Data":"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"}
Sep 09 20:11:05.373475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:05.373439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerStarted","Data":"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"}
Sep 09 20:11:05.373980 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:05.373512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerStarted","Data":"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"}
Sep 09 20:11:05.373980 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:05.373820    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:11:05.393722 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:05.393663    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podStartSLOduration=2.721694488 podStartE2EDuration="6.393650235s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.671955747s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:05.391427531 +0000 UTC m=+954.792104722" watchObservedRunningTime="2026-09-09 20:11:05.393650235 +0000 UTC m=+954.794327426"
Sep 09 20:11:06.379911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:06.379874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:11:06.381126 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:06.381101    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:07.384824 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:07.384785    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:12.389212 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:12.389182    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:11:12.389707 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:12.389658    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:22.390419 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:22.390365    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:32.390467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:32.390427    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:42.390242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:42.390197    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:52.390440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:52.390396    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:11:58.750297 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:58.750265    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-82b2z_9ecd1c18-7540-414e-bece-7d03fadcb8d4/kserve-container/0.log"
Sep 09 20:11:58.948184 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:58.948150    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:11:58.948422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:58.948397    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kserve-container" containerID="cri-o://fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305" gracePeriod=30
Sep 09 20:11:58.948569 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:58.948411    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kube-rbac-proxy" containerID="cri-o://0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c" gracePeriod=30
Sep 09 20:11:59.192740 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.192717    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:11:59.281268 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.281228    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls\") pod \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") "
Sep 09 20:11:59.281433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.281289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb75z\" (UniqueName: \"kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z\") pod \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") "
Sep 09 20:11:59.281433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.281374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config\") pod \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\" (UID: \"9ecd1c18-7540-414e-bece-7d03fadcb8d4\") "
Sep 09 20:11:59.281716 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.281662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ecd1c18-7540-414e-bece-7d03fadcb8d4" (UID: "9ecd1c18-7540-414e-bece-7d03fadcb8d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:59.283434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.283400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ecd1c18-7540-414e-bece-7d03fadcb8d4" (UID: "9ecd1c18-7540-414e-bece-7d03fadcb8d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:59.283434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.283420    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z" (OuterVolumeSpecName: "kube-api-access-rb75z") pod "9ecd1c18-7540-414e-bece-7d03fadcb8d4" (UID: "9ecd1c18-7540-414e-bece-7d03fadcb8d4"). InnerVolumeSpecName "kube-api-access-rb75z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:59.382030 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.381955    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:11:59.382030 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.381982    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecd1c18-7540-414e-bece-7d03fadcb8d4-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:11:59.382030 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.381993    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb75z\" (UniqueName: \"kubernetes.io/projected/9ecd1c18-7540-414e-bece-7d03fadcb8d4-kube-api-access-rb75z\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:11:59.631987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.631952    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerID="0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c" exitCode=2
Sep 09 20:11:59.631987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.631980    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerID="fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305" exitCode=2
Sep 09 20:11:59.632202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.632001    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"
Sep 09 20:11:59.632202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.632033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerDied","Data":"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"}
Sep 09 20:11:59.632202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.632101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerDied","Data":"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"}
Sep 09 20:11:59.632202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.632135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z" event={"ID":"9ecd1c18-7540-414e-bece-7d03fadcb8d4","Type":"ContainerDied","Data":"11d5302e195da1e2647819b9ce9c68302242e94341fb7703f40cd5d478a0133f"}
Sep 09 20:11:59.632202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.632165    2582 scope.go:117] "RemoveContainer" containerID="0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"
Sep 09 20:11:59.641107 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.641090    2582 scope.go:117] "RemoveContainer" containerID="fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"
Sep 09 20:11:59.648404 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.648387    2582 scope.go:117] "RemoveContainer" containerID="0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"
Sep 09 20:11:59.648656 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:11:59.648638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c\": container with ID starting with 0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c not found: ID does not exist" containerID="0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"
Sep 09 20:11:59.648756 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.648664    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"} err="failed to get container status \"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c\": rpc error: code = NotFound desc = could not find container \"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c\": container with ID starting with 0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c not found: ID does not exist"
Sep 09 20:11:59.648756 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.648691    2582 scope.go:117] "RemoveContainer" containerID="fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"
Sep 09 20:11:59.648932 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:11:59.648915    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305\": container with ID starting with fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305 not found: ID does not exist" containerID="fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"
Sep 09 20:11:59.648979 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.648936    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"} err="failed to get container status \"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305\": rpc error: code = NotFound desc = could not find container \"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305\": container with ID starting with fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305 not found: ID does not exist"
Sep 09 20:11:59.648979 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.648949    2582 scope.go:117] "RemoveContainer" containerID="0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"
Sep 09 20:11:59.649190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.649170    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c"} err="failed to get container status \"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c\": rpc error: code = NotFound desc = could not find container \"0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c\": container with ID starting with 0ac1455b3e9846934ae4f34d9aeb9e11acb2466d74f9c572f3d0f0e345d44b8c not found: ID does not exist"
Sep 09 20:11:59.649238 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.649193    2582 scope.go:117] "RemoveContainer" containerID="fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"
Sep 09 20:11:59.649428 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.649410    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305"} err="failed to get container status \"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305\": rpc error: code = NotFound desc = could not find container \"fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305\": container with ID starting with fa2e39538bf5f4418d966285661f410f35b22a3fd6ea80586d3d3286197a1305 not found: ID does not exist"
Sep 09 20:11:59.653379 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.653359    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:11:59.656881 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:11:59.656858    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-82b2z"]
Sep 09 20:12:01.130317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:01.130278    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" path="/var/lib/kubelet/pods/9ecd1c18-7540-414e-bece-7d03fadcb8d4/volumes"
Sep 09 20:12:02.390532 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:02.390488    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:12:12.390312 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:12.390261    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:12:22.119854 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:22.119817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:12:29.213427 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:29.213394    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:12:29.213954 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:29.213660    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" containerID="cri-o://9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45" gracePeriod=30
Sep 09 20:12:29.213954 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:29.213716    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kube-rbac-proxy" containerID="cri-o://59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117" gracePeriod=30
Sep 09 20:12:29.772496 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:29.772463    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerID="59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117" exitCode=2
Sep 09 20:12:29.772714 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:29.772532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerDied","Data":"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"}
Sep 09 20:12:32.120032 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:32.119992    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 09 20:12:32.385445 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:32.385355    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 09 20:12:34.359548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.359520    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:12:34.416521 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416488    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:12:34.416851 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416837    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kserve-container"
Sep 09 20:12:34.416851 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416851    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kserve-container"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416869    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kube-rbac-proxy"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kube-rbac-proxy"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416889    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="storage-initializer"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416897    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="storage-initializer"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416902    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kube-rbac-proxy"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416907    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kube-rbac-proxy"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416916    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container"
Sep 09 20:12:34.416943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container"
Sep 09 20:12:34.417198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416975    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kserve-container"
Sep 09 20:12:34.417198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416983    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kserve-container"
Sep 09 20:12:34.417198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416990    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerName="kube-rbac-proxy"
Sep 09 20:12:34.417198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.416997    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecd1c18-7540-414e-bece-7d03fadcb8d4" containerName="kube-rbac-proxy"
Sep 09 20:12:34.420170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.420151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.422563 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.422536    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 09 20:12:34.422656 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.422612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:12:34.431789 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.431765    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:12:34.474672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.474643    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjltv\" (UniqueName: \"kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv\") pod \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") "
Sep 09 20:12:34.474822 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.474765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert\") pod \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") "
Sep 09 20:12:34.474871 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.474824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls\") pod \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") "
Sep 09 20:12:34.474871 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.474860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location\") pod \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") "
Sep 09 20:12:34.474968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.474894    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config\") pod \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\" (UID: \"b9ff8ecf-68c9-4266-9b52-ccbec3607cca\") "
Sep 09 20:12:34.475211 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.475182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9ff8ecf-68c9-4266-9b52-ccbec3607cca" (UID: "b9ff8ecf-68c9-4266-9b52-ccbec3607cca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:34.475291 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.475253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9ff8ecf-68c9-4266-9b52-ccbec3607cca" (UID: "b9ff8ecf-68c9-4266-9b52-ccbec3607cca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:34.475327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.475290    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9ff8ecf-68c9-4266-9b52-ccbec3607cca" (UID: "b9ff8ecf-68c9-4266-9b52-ccbec3607cca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:34.476887 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.476843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv" (OuterVolumeSpecName: "kube-api-access-wjltv") pod "b9ff8ecf-68c9-4266-9b52-ccbec3607cca" (UID: "b9ff8ecf-68c9-4266-9b52-ccbec3607cca"). InnerVolumeSpecName "kube-api-access-wjltv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:34.476975 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.476953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9ff8ecf-68c9-4266-9b52-ccbec3607cca" (UID: "b9ff8ecf-68c9-4266-9b52-ccbec3607cca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:34.576015 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.575978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggbz\" (UniqueName: \"kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.576169 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.576169 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.576169 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576229    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576241    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576249    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576258    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.576277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.576267    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjltv\" (UniqueName: \"kubernetes.io/projected/b9ff8ecf-68c9-4266-9b52-ccbec3607cca-kube-api-access-wjltv\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.677003 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.676967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677125 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677125 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677125 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677125 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggbz\" (UniqueName: \"kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677537 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677693 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.677746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.677674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.679514 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.679496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.685755 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.685727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggbz\" (UniqueName: \"kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.731454 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.731385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:34.806718 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.806671    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" containerID="9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45" exitCode=0
Sep 09 20:12:34.806839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.806713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerDied","Data":"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"}
Sep 09 20:12:34.806839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.806773    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"
Sep 09 20:12:34.806839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.806796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r" event={"ID":"b9ff8ecf-68c9-4266-9b52-ccbec3607cca","Type":"ContainerDied","Data":"3f5838f1340ab9430a9490b6bc43245a4d2ad8ef1891a63fec9ab345e2cb6a8b"}
Sep 09 20:12:34.806839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.806824    2582 scope.go:117] "RemoveContainer" containerID="59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"
Sep 09 20:12:34.815042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.815022    2582 scope.go:117] "RemoveContainer" containerID="9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"
Sep 09 20:12:34.822643 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.822628    2582 scope.go:117] "RemoveContainer" containerID="3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"
Sep 09 20:12:34.830159 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.830142    2582 scope.go:117] "RemoveContainer" containerID="59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"
Sep 09 20:12:34.830429 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:12:34.830411    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117\": container with ID starting with 59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117 not found: ID does not exist" containerID="59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"
Sep 09 20:12:34.830478 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.830437    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117"} err="failed to get container status \"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117\": rpc error: code = NotFound desc = could not find container \"59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117\": container with ID starting with 59b82ad01683ea2322dd6aa23f4de1bdf8f0e7ff52d0341f095e7ce74c608117 not found: ID does not exist"
Sep 09 20:12:34.830478 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.830453    2582 scope.go:117] "RemoveContainer" containerID="9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"
Sep 09 20:12:34.831240 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:12:34.831062    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45\": container with ID starting with 9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45 not found: ID does not exist" containerID="9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"
Sep 09 20:12:34.831240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.831124    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45"} err="failed to get container status \"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45\": rpc error: code = NotFound desc = could not find container \"9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45\": container with ID starting with 9feb2540538f74996a8c88fc99f9f22178db7136209bf67c8bba4d9b30c42e45 not found: ID does not exist"
Sep 09 20:12:34.831240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.831145    2582 scope.go:117] "RemoveContainer" containerID="3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"
Sep 09 20:12:34.831629 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:12:34.831608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010\": container with ID starting with 3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010 not found: ID does not exist" containerID="3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"
Sep 09 20:12:34.831717 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.831633    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010"} err="failed to get container status \"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010\": rpc error: code = NotFound desc = could not find container \"3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010\": container with ID starting with 3ba3007b23d9a71caab312d1e53c97f6f7cb1f53c848185ef953f03adef3b010 not found: ID does not exist"
Sep 09 20:12:34.833215 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.833196    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:12:34.840159 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:34.840132    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-nj99r"]
Sep 09 20:12:35.002019 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:35.001875    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:12:35.004647 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:12:35.004620    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb0fa4_aa04_4e47_95f6_b0ede4d9511d.slice/crio-d40586ed275133d2cd70f218a57e3b3fcaa2085323de5492c5b3de18ba7b11b2 WatchSource:0}: Error finding container d40586ed275133d2cd70f218a57e3b3fcaa2085323de5492c5b3de18ba7b11b2: Status 404 returned error can't find the container with id d40586ed275133d2cd70f218a57e3b3fcaa2085323de5492c5b3de18ba7b11b2
Sep 09 20:12:35.124020 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:35.123986    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ff8ecf-68c9-4266-9b52-ccbec3607cca" path="/var/lib/kubelet/pods/b9ff8ecf-68c9-4266-9b52-ccbec3607cca/volumes"
Sep 09 20:12:35.812074 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:35.812031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerStarted","Data":"d1e9bdbe83c993c19710f904acf6c8ac98e9c16d2f13a1996c186bb0d4a29c50"}
Sep 09 20:12:35.812506 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:35.812109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerStarted","Data":"d40586ed275133d2cd70f218a57e3b3fcaa2085323de5492c5b3de18ba7b11b2"}
Sep 09 20:12:39.831118 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:39.831082    2582 generic.go:356] "Generic (PLEG): container finished" podID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerID="d1e9bdbe83c993c19710f904acf6c8ac98e9c16d2f13a1996c186bb0d4a29c50" exitCode=0
Sep 09 20:12:39.831517 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:39.831155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerDied","Data":"d1e9bdbe83c993c19710f904acf6c8ac98e9c16d2f13a1996c186bb0d4a29c50"}
Sep 09 20:12:40.837555 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:40.837524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerStarted","Data":"13b666c5c32bb1d80e4f6eaa77cfc08e33d363a406d9b83c9884a58d96a8e438"}
Sep 09 20:12:40.837961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:40.837588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerStarted","Data":"4a128278991acf4a6b1292d0621b701cf81efec4c641ba9a0c0c51c30322b598"}
Sep 09 20:12:40.837961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:40.837782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:40.859930 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:40.859859    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podStartSLOduration=2.877894384 podStartE2EDuration="6.859843397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981949013s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:40.857552559 +0000 UTC m=+1050.258229752" watchObservedRunningTime="2026-09-09 20:12:40.859843397 +0000 UTC m=+1050.260520590"
Sep 09 20:12:41.843267 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:41.843221    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:12:47.853765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:12:47.853731    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:13:17.854987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:13:17.854943    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:13:27.854848 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:13:27.854803    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:13:37.854377 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:13:37.854337    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:13:47.854363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:13:47.854323    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:13:57.876558 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:13:57.876522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:14:04.519650 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:04.519613    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:14:04.520084 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:04.519882    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" containerID="cri-o://4a128278991acf4a6b1292d0621b701cf81efec4c641ba9a0c0c51c30322b598" gracePeriod=30
Sep 09 20:14:04.520084 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:04.519911    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kube-rbac-proxy" containerID="cri-o://13b666c5c32bb1d80e4f6eaa77cfc08e33d363a406d9b83c9884a58d96a8e438" gracePeriod=30
Sep 09 20:14:05.216928 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:05.216885    2582 generic.go:356] "Generic (PLEG): container finished" podID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerID="13b666c5c32bb1d80e4f6eaa77cfc08e33d363a406d9b83c9884a58d96a8e438" exitCode=2
Sep 09 20:14:05.217091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:05.216957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerDied","Data":"13b666c5c32bb1d80e4f6eaa77cfc08e33d363a406d9b83c9884a58d96a8e438"}
Sep 09 20:14:07.848401 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:07.848358    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 09 20:14:07.854826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:07.854797    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 09 20:14:09.237732 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.237696    2582 generic.go:356] "Generic (PLEG): container finished" podID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerID="4a128278991acf4a6b1292d0621b701cf81efec4c641ba9a0c0c51c30322b598" exitCode=0
Sep 09 20:14:09.238094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.237709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerDied","Data":"4a128278991acf4a6b1292d0621b701cf81efec4c641ba9a0c0c51c30322b598"}
Sep 09 20:14:09.271026 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.270997    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:14:09.358670 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert\") pod \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") "
Sep 09 20:14:09.358670 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location\") pod \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") "
Sep 09 20:14:09.358872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358690    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggbz\" (UniqueName: \"kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz\") pod \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") "
Sep 09 20:14:09.358913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config\") pod \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") "
Sep 09 20:14:09.358967 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358940    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" (UID: "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:09.359029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.358948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls\") pod \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\" (UID: \"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d\") "
Sep 09 20:14:09.359029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.359017    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" (UID: "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:09.359267 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.359246    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.359267 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.359268    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.359407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.359250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" (UID: "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:09.360650 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.360631    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz" (OuterVolumeSpecName: "kube-api-access-fggbz") pod "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" (UID: "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d"). InnerVolumeSpecName "kube-api-access-fggbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:09.360883 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.360823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" (UID: "90bb0fa4-aa04-4e47-95f6-b0ede4d9511d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:09.460448 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.460418    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggbz\" (UniqueName: \"kubernetes.io/projected/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-api-access-fggbz\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.460448 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.460443    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.460605 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.460456    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.703240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703203    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:14:09.703558 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703545    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="storage-initializer"
Sep 09 20:14:09.703606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703560    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="storage-initializer"
Sep 09 20:14:09.703606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703572    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container"
Sep 09 20:14:09.703606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703580    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container"
Sep 09 20:14:09.703606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703602    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kube-rbac-proxy"
Sep 09 20:14:09.703746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703609    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kube-rbac-proxy"
Sep 09 20:14:09.703746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703657    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kube-rbac-proxy"
Sep 09 20:14:09.703746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.703667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" containerName="kserve-container"
Sep 09 20:14:09.706894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.706878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.709217 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.709182    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 09 20:14:09.709345 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.709229    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:14:09.715556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.715534    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:14:09.763838 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.763806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.763838 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.763842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.764022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.763864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnb8\" (UniqueName: \"kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.764022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.763963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.764022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.764005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865083 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865083 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865295 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnb8\" (UniqueName: \"kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865295 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865295 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865295 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:14:09.865284    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 09 20:14:09.865514 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:14:09.865349    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls podName:91c06839-56bd-457e-a856-28bd80647b92 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:10.365329786 +0000 UTC m=+1139.766006958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" (UID: "91c06839-56bd-457e-a856-28bd80647b92") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 09 20:14:09.865514 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865758 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.865941 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.865919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:09.875312 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:09.875291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnb8\" (UniqueName: \"kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:10.246662 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.246631    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"
Sep 09 20:14:10.246662 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.246643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t" event={"ID":"90bb0fa4-aa04-4e47-95f6-b0ede4d9511d","Type":"ContainerDied","Data":"d40586ed275133d2cd70f218a57e3b3fcaa2085323de5492c5b3de18ba7b11b2"}
Sep 09 20:14:10.247117 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.246728    2582 scope.go:117] "RemoveContainer" containerID="13b666c5c32bb1d80e4f6eaa77cfc08e33d363a406d9b83c9884a58d96a8e438"
Sep 09 20:14:10.255332 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.255315    2582 scope.go:117] "RemoveContainer" containerID="4a128278991acf4a6b1292d0621b701cf81efec4c641ba9a0c0c51c30322b598"
Sep 09 20:14:10.262711 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.262694    2582 scope.go:117] "RemoveContainer" containerID="d1e9bdbe83c993c19710f904acf6c8ac98e9c16d2f13a1996c186bb0d4a29c50"
Sep 09 20:14:10.270862 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.270818    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:14:10.274814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.274792    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-fmq6t"]
Sep 09 20:14:10.369199 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.369167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:10.371569 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.371540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:10.618262 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.618163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:10.890449 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:10.890376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:14:10.893562 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:14:10.893533    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c06839_56bd_457e_a856_28bd80647b92.slice/crio-e6f185258986a2e50fd7b941de508fa79151a0ba331ed0312410e14c13633e28 WatchSource:0}: Error finding container e6f185258986a2e50fd7b941de508fa79151a0ba331ed0312410e14c13633e28: Status 404 returned error can't find the container with id e6f185258986a2e50fd7b941de508fa79151a0ba331ed0312410e14c13633e28
Sep 09 20:14:11.125518 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:11.125469    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90bb0fa4-aa04-4e47-95f6-b0ede4d9511d" path="/var/lib/kubelet/pods/90bb0fa4-aa04-4e47-95f6-b0ede4d9511d/volumes"
Sep 09 20:14:11.254133 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:11.254094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerStarted","Data":"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"}
Sep 09 20:14:11.254592 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:11.254152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerStarted","Data":"e6f185258986a2e50fd7b941de508fa79151a0ba331ed0312410e14c13633e28"}
Sep 09 20:14:15.273334 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:15.273296    2582 generic.go:356] "Generic (PLEG): container finished" podID="91c06839-56bd-457e-a856-28bd80647b92" containerID="34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264" exitCode=0
Sep 09 20:14:15.273719 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:15.273369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerDied","Data":"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"}
Sep 09 20:14:16.280063 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:16.280028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerStarted","Data":"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"}
Sep 09 20:14:16.280425 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:16.280092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerStarted","Data":"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"}
Sep 09 20:14:16.280425 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:16.280313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:16.301051 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:16.301006    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podStartSLOduration=3.441281334 podStartE2EDuration="7.300991589s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859710255s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:16.299249204 +0000 UTC m=+1145.699926397" watchObservedRunningTime="2026-09-09 20:14:16.300991589 +0000 UTC m=+1145.701668782"
Sep 09 20:14:17.285330 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:17.285298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:23.295367 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:23.295339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:14:53.296775 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:14:53.296736    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:15:03.296463 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:03.296415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:15:13.296490 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:13.296449    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:15:23.295965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:23.295925    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:15:32.177792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:32.177761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:15:39.817837 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:39.817802    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:15:39.818303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:39.818053    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" containerID="cri-o://c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44" gracePeriod=30
Sep 09 20:15:39.818303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:39.818105    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kube-rbac-proxy" containerID="cri-o://15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb" gracePeriod=30
Sep 09 20:15:40.677601 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:40.677565    2582 generic.go:356] "Generic (PLEG): container finished" podID="91c06839-56bd-457e-a856-28bd80647b92" containerID="15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb" exitCode=2
Sep 09 20:15:40.677787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:40.677636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerDied","Data":"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"}
Sep 09 20:15:42.120049 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:42.120000    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 09 20:15:43.291203 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:43.291152    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 09 20:15:44.360758 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.360733    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:15:44.461970 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.461940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwnb8\" (UniqueName: \"kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8\") pod \"91c06839-56bd-457e-a856-28bd80647b92\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") "
Sep 09 20:15:44.462134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.461981    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert\") pod \"91c06839-56bd-457e-a856-28bd80647b92\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") "
Sep 09 20:15:44.462134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config\") pod \"91c06839-56bd-457e-a856-28bd80647b92\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") "
Sep 09 20:15:44.462134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") pod \"91c06839-56bd-457e-a856-28bd80647b92\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") "
Sep 09 20:15:44.462134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location\") pod \"91c06839-56bd-457e-a856-28bd80647b92\" (UID: \"91c06839-56bd-457e-a856-28bd80647b92\") "
Sep 09 20:15:44.462426 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462403    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91c06839-56bd-457e-a856-28bd80647b92" (UID: "91c06839-56bd-457e-a856-28bd80647b92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:44.462481 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91c06839-56bd-457e-a856-28bd80647b92" (UID: "91c06839-56bd-457e-a856-28bd80647b92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:44.462481 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.462445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91c06839-56bd-457e-a856-28bd80647b92" (UID: "91c06839-56bd-457e-a856-28bd80647b92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:44.464070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.464046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8" (OuterVolumeSpecName: "kube-api-access-jwnb8") pod "91c06839-56bd-457e-a856-28bd80647b92" (UID: "91c06839-56bd-457e-a856-28bd80647b92"). InnerVolumeSpecName "kube-api-access-jwnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:44.464175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.464127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91c06839-56bd-457e-a856-28bd80647b92" (UID: "91c06839-56bd-457e-a856-28bd80647b92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:15:44.563324 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.563292    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91c06839-56bd-457e-a856-28bd80647b92-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:15:44.563324 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.563319    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91c06839-56bd-457e-a856-28bd80647b92-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:15:44.563324 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.563328    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnb8\" (UniqueName: \"kubernetes.io/projected/91c06839-56bd-457e-a856-28bd80647b92-kube-api-access-jwnb8\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:15:44.563512 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.563337    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:15:44.563512 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.563346    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91c06839-56bd-457e-a856-28bd80647b92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:15:44.700033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.699997    2582 generic.go:356] "Generic (PLEG): container finished" podID="91c06839-56bd-457e-a856-28bd80647b92" containerID="c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44" exitCode=0
Sep 09 20:15:44.700187 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.700054    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"
Sep 09 20:15:44.700187 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.700059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerDied","Data":"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"}
Sep 09 20:15:44.700261 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.700246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd" event={"ID":"91c06839-56bd-457e-a856-28bd80647b92","Type":"ContainerDied","Data":"e6f185258986a2e50fd7b941de508fa79151a0ba331ed0312410e14c13633e28"}
Sep 09 20:15:44.700294 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.700287    2582 scope.go:117] "RemoveContainer" containerID="15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"
Sep 09 20:15:44.709773 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.709755    2582 scope.go:117] "RemoveContainer" containerID="c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"
Sep 09 20:15:44.717403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.717386    2582 scope.go:117] "RemoveContainer" containerID="34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"
Sep 09 20:15:44.724325 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.724297    2582 scope.go:117] "RemoveContainer" containerID="15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"
Sep 09 20:15:44.724578 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:15:44.724561    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb\": container with ID starting with 15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb not found: ID does not exist" containerID="15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"
Sep 09 20:15:44.724630 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.724590    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb"} err="failed to get container status \"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb\": rpc error: code = NotFound desc = could not find container \"15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb\": container with ID starting with 15be556830e2ea4f94715ecdfa6489e79a56108eb61280cfa01eb0ebb4a9b4fb not found: ID does not exist"
Sep 09 20:15:44.724630 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.724607    2582 scope.go:117] "RemoveContainer" containerID="c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"
Sep 09 20:15:44.724890 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:15:44.724873    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44\": container with ID starting with c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44 not found: ID does not exist" containerID="c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"
Sep 09 20:15:44.724936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.724896    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44"} err="failed to get container status \"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44\": rpc error: code = NotFound desc = could not find container \"c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44\": container with ID starting with c8e29b76ff7fc14d5cd9fe258fa9d01be65566dd69d87c13df7456c7dd921f44 not found: ID does not exist"
Sep 09 20:15:44.724936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.724921    2582 scope.go:117] "RemoveContainer" containerID="34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"
Sep 09 20:15:44.725149 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:15:44.725133    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264\": container with ID starting with 34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264 not found: ID does not exist" containerID="34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"
Sep 09 20:15:44.725199 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.725152    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264"} err="failed to get container status \"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264\": rpc error: code = NotFound desc = could not find container \"34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264\": container with ID starting with 34939f6c91a17a3fe19bae5e0a80ee53db5eb234bd4397c44db4095d3c236264 not found: ID does not exist"
Sep 09 20:15:44.728402 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.728381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:15:44.734468 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:44.734447    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gf6wd"]
Sep 09 20:15:45.045365 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045288    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:15:45.045712 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045697    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kube-rbac-proxy"
Sep 09 20:15:45.045764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kube-rbac-proxy"
Sep 09 20:15:45.045764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045724    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="storage-initializer"
Sep 09 20:15:45.045764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045729    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="storage-initializer"
Sep 09 20:15:45.045764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045738    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container"
Sep 09 20:15:45.045764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045743    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container"
Sep 09 20:15:45.045914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045812    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kube-rbac-proxy"
Sep 09 20:15:45.045914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.045827    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c06839-56bd-457e-a856-28bd80647b92" containerName="kserve-container"
Sep 09 20:15:45.051620 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.051598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.054127 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:15:45.054242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054196    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:15:45.054583 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054568    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:15:45.054984 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:15:45.054984 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:15:45.055145 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.054970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:15:45.055289 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.055274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 09 20:15:45.063903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.063882    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:15:45.068559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.068538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.068657 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.068578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.068657 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.068610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.068795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.068728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89jw\" (UniqueName: \"kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.068857 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.068805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.130555 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.130516    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c06839-56bd-457e-a856-28bd80647b92" path="/var/lib/kubelet/pods/91c06839-56bd-457e-a856-28bd80647b92/volumes"
Sep 09 20:15:45.169466 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89jw\" (UniqueName: \"kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.169638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.169638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.169638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.169638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.169910 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:15:45.169658    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 20:15:45.169910 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:15:45.169741    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls podName:c6c8371c-6569-4cbb-8bd9-acb787d31461 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:45.669720764 +0000 UTC m=+1235.070397937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 20:15:45.170022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.169952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.170197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.170177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.170249 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.170230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.183611 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.183586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89jw\" (UniqueName: \"kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.673872 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.673826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.676343 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.676317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:45.962563 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:45.962536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:46.237939 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:46.237915    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:15:46.240246 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:15:46.240218    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c8371c_6569_4cbb_8bd9_acb787d31461.slice/crio-753b35fa15fbf1316ba57b00153cea4eebf1ec37fda01f1cd2714f16af71da47 WatchSource:0}: Error finding container 753b35fa15fbf1316ba57b00153cea4eebf1ec37fda01f1cd2714f16af71da47: Status 404 returned error can't find the container with id 753b35fa15fbf1316ba57b00153cea4eebf1ec37fda01f1cd2714f16af71da47
Sep 09 20:15:46.714820 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:46.714785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerStarted","Data":"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"}
Sep 09 20:15:46.715202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:46.714841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerStarted","Data":"753b35fa15fbf1316ba57b00153cea4eebf1ec37fda01f1cd2714f16af71da47"}
Sep 09 20:15:50.736072 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:50.736038    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerID="29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331" exitCode=0
Sep 09 20:15:50.736429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:50.736095    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerDied","Data":"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"}
Sep 09 20:15:51.743248 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:51.743214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerStarted","Data":"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"}
Sep 09 20:15:51.745572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:51.743302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerStarted","Data":"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"}
Sep 09 20:15:51.745572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:51.743505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:51.745572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:51.743656    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:15:51.766250 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:51.766205    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podStartSLOduration=2.763558165 podStartE2EDuration="6.766192638s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.002634473s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:51.763941708 +0000 UTC m=+1241.164618899" watchObservedRunningTime="2026-09-09 20:15:51.766192638 +0000 UTC m=+1241.166869829"
Sep 09 20:15:57.753211 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:15:57.753180    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:16:27.754234 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:16:27.754192    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:16:37.753805 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:16:37.753763    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:16:47.754332 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:16:47.754285    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:16:57.754216 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:16:57.754177    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:17:07.777616 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:07.777575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:17:15.113070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:15.113035    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:17:15.113628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:15.113345    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" containerID="cri-o://0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa" gracePeriod=30
Sep 09 20:17:15.113628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:15.113405    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kube-rbac-proxy" containerID="cri-o://5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c" gracePeriod=30
Sep 09 20:17:16.126982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:16.126945    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerID="5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c" exitCode=2
Sep 09 20:17:16.127356 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:16.127016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerDied","Data":"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"}
Sep 09 20:17:17.748370 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:17.748330    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 09 20:17:17.754765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:17.754727    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 09 20:17:20.043862 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.043836    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:17:20.146025 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.145992    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerID="0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa" exitCode=0
Sep 09 20:17:20.146175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.146034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerDied","Data":"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"}
Sep 09 20:17:20.146175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.146042    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"
Sep 09 20:17:20.146175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.146086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz" event={"ID":"c6c8371c-6569-4cbb-8bd9-acb787d31461","Type":"ContainerDied","Data":"753b35fa15fbf1316ba57b00153cea4eebf1ec37fda01f1cd2714f16af71da47"}
Sep 09 20:17:20.146175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.146112    2582 scope.go:117] "RemoveContainer" containerID="5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"
Sep 09 20:17:20.154670 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.154653    2582 scope.go:117] "RemoveContainer" containerID="0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"
Sep 09 20:17:20.161903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.161887    2582 scope.go:117] "RemoveContainer" containerID="29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"
Sep 09 20:17:20.168951 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.168936    2582 scope.go:117] "RemoveContainer" containerID="5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"
Sep 09 20:17:20.169213 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:17:20.169193    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c\": container with ID starting with 5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c not found: ID does not exist" containerID="5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"
Sep 09 20:17:20.169273 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.169221    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c"} err="failed to get container status \"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c\": rpc error: code = NotFound desc = could not find container \"5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c\": container with ID starting with 5645d0c44c053e364d5147f93ebdf705762cbd002f820f64b62aff290f1cf52c not found: ID does not exist"
Sep 09 20:17:20.169273 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.169238    2582 scope.go:117] "RemoveContainer" containerID="0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"
Sep 09 20:17:20.169470 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:17:20.169456    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa\": container with ID starting with 0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa not found: ID does not exist" containerID="0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"
Sep 09 20:17:20.169533 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.169474    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa"} err="failed to get container status \"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa\": rpc error: code = NotFound desc = could not find container \"0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa\": container with ID starting with 0588513f98e49778b691a97e4cded1da559cf0e31237d34ecb6f3800b6dc65aa not found: ID does not exist"
Sep 09 20:17:20.169533 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.169489    2582 scope.go:117] "RemoveContainer" containerID="29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"
Sep 09 20:17:20.170036 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:17:20.170005    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331\": container with ID starting with 29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331 not found: ID does not exist" containerID="29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"
Sep 09 20:17:20.170129 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.170042    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331"} err="failed to get container status \"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331\": rpc error: code = NotFound desc = could not find container \"29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331\": container with ID starting with 29b01d756f89ab23e39e7c5d389ebef3d70e679a7b286322ea289f2a29c34331 not found: ID does not exist"
Sep 09 20:17:20.177463 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert\") pod \"c6c8371c-6569-4cbb-8bd9-acb787d31461\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") "
Sep 09 20:17:20.177572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177463    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") pod \"c6c8371c-6569-4cbb-8bd9-acb787d31461\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") "
Sep 09 20:17:20.177572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config\") pod \"c6c8371c-6569-4cbb-8bd9-acb787d31461\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") "
Sep 09 20:17:20.177572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177564    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location\") pod \"c6c8371c-6569-4cbb-8bd9-acb787d31461\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") "
Sep 09 20:17:20.177757 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c89jw\" (UniqueName: \"kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw\") pod \"c6c8371c-6569-4cbb-8bd9-acb787d31461\" (UID: \"c6c8371c-6569-4cbb-8bd9-acb787d31461\") "
Sep 09 20:17:20.177873 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6c8371c-6569-4cbb-8bd9-acb787d31461" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:20.178011 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.177980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6c8371c-6569-4cbb-8bd9-acb787d31461" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:20.178011 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.178002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6c8371c-6569-4cbb-8bd9-acb787d31461" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:20.179536 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.179507    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6c8371c-6569-4cbb-8bd9-acb787d31461" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:20.179775 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.179747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw" (OuterVolumeSpecName: "kube-api-access-c89jw") pod "c6c8371c-6569-4cbb-8bd9-acb787d31461" (UID: "c6c8371c-6569-4cbb-8bd9-acb787d31461"). InnerVolumeSpecName "kube-api-access-c89jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:20.278857 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.278819    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89jw\" (UniqueName: \"kubernetes.io/projected/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-api-access-c89jw\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:17:20.278857 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.278852    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:17:20.279080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.278866    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6c8371c-6569-4cbb-8bd9-acb787d31461-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:17:20.279080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.278880    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6c8371c-6569-4cbb-8bd9-acb787d31461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:17:20.279080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.278892    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6c8371c-6569-4cbb-8bd9-acb787d31461-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:17:20.471319 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.471291    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:17:20.476325 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:20.476301    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jh5wz"]
Sep 09 20:17:21.124881 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:17:21.124848    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" path="/var/lib/kubelet/pods/c6c8371c-6569-4cbb-8bd9-acb787d31461/volumes"
Sep 09 20:19:19.476562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476526    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476883    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="storage-initializer"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476896    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="storage-initializer"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476904    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476910    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476916    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kube-rbac-proxy"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476922    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kube-rbac-proxy"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.476988    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kserve-container"
Sep 09 20:19:19.477024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.477000    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c8371c-6569-4cbb-8bd9-acb787d31461" containerName="kube-rbac-proxy"
Sep 09 20:19:19.480264 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.480240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.483299 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.483277    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 09 20:19:19.483633 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.483617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:19:19.484294 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.484272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:19:19.484394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.484294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:19:19.484394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.484294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:19:19.484394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.484272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:19:19.484617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.484594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:19:19.501045 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.501024    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:19:19.558223 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.558192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.558422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.558231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.558422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.558249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.558422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.558374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7qq\" (UniqueName: \"kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.558561 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.558426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.659455 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7qq\" (UniqueName: \"kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.659455 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.659674 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.659818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.659818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.660050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.659914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.660378 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.660355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.660518 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.660497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.662298 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.662280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.673341 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.673321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7qq\" (UniqueName: \"kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:19.790549 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:19.790474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:19:20.064558 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:20.064529    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:19:20.066481 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:19:20.066448    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda493e682_6779_4eee_906c_53688381db34.slice/crio-76de79395af13bbbd9bc93ad5f0cea6740a6b2bf6c3c65e550969e0cca79dbb0 WatchSource:0}: Error finding container 76de79395af13bbbd9bc93ad5f0cea6740a6b2bf6c3c65e550969e0cca79dbb0: Status 404 returned error can't find the container with id 76de79395af13bbbd9bc93ad5f0cea6740a6b2bf6c3c65e550969e0cca79dbb0
Sep 09 20:19:20.688377 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:20.688339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerStarted","Data":"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"}
Sep 09 20:19:20.688816 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:20.688401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerStarted","Data":"76de79395af13bbbd9bc93ad5f0cea6740a6b2bf6c3c65e550969e0cca79dbb0"}
Sep 09 20:19:23.703070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:23.703031    2582 generic.go:356] "Generic (PLEG): container finished" podID="a493e682-6779-4eee-906c-53688381db34" containerID="fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c" exitCode=0
Sep 09 20:19:23.703444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:23.703084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerDied","Data":"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"}
Sep 09 20:19:23.704566 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:19:23.704546    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:21:41.432268 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:41.432233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerStarted","Data":"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"}
Sep 09 20:21:41.432667 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:41.432292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerStarted","Data":"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"}
Sep 09 20:21:41.432667 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:41.432431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:21:41.432667 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:41.432479    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:21:41.457104 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:41.457058    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" podStartSLOduration=2.07420093 podStartE2EDuration="2m22.457033682s" totalImagesPullingTime="2m16.856919784s" totalInitContainerRuntime="3.525912968s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:41.455661325 +0000 UTC m=+1590.856338516" watchObservedRunningTime="2026-09-09 20:21:41.457033682 +0000 UTC m=+1590.857710873"
Sep 09 20:21:47.442608 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:21:47.442577    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:22:17.473924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:17.473875    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:22:27.445884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:27.445855    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:22:29.612938 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:29.612898    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:22:29.613372 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:29.613304    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kserve-container" containerID="cri-o://f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e" gracePeriod=30
Sep 09 20:22:29.614634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:29.614571    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kube-rbac-proxy" containerID="cri-o://e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a" gracePeriod=30
Sep 09 20:22:30.664839 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:30.664806    2582 generic.go:356] "Generic (PLEG): container finished" podID="a493e682-6779-4eee-906c-53688381db34" containerID="e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a" exitCode=2
Sep 09 20:22:30.665197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:30.664878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerDied","Data":"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"}
Sep 09 20:22:32.437455 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:32.437418    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 09 20:22:36.663474 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.663450    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:22:36.696420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.696386    2582 generic.go:356] "Generic (PLEG): container finished" podID="a493e682-6779-4eee-906c-53688381db34" containerID="f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e" exitCode=0
Sep 09 20:22:36.696573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.696435    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"
Sep 09 20:22:36.696573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.696465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerDied","Data":"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"}
Sep 09 20:22:36.696573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.696567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54" event={"ID":"a493e682-6779-4eee-906c-53688381db34","Type":"ContainerDied","Data":"76de79395af13bbbd9bc93ad5f0cea6740a6b2bf6c3c65e550969e0cca79dbb0"}
Sep 09 20:22:36.696775 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.696608    2582 scope.go:117] "RemoveContainer" containerID="e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"
Sep 09 20:22:36.706536 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.706521    2582 scope.go:117] "RemoveContainer" containerID="f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"
Sep 09 20:22:36.715955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.715937    2582 scope.go:117] "RemoveContainer" containerID="fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"
Sep 09 20:22:36.723118 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723101    2582 scope.go:117] "RemoveContainer" containerID="e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"
Sep 09 20:22:36.723384 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:22:36.723367    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a\": container with ID starting with e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a not found: ID does not exist" containerID="e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"
Sep 09 20:22:36.723434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723392    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a"} err="failed to get container status \"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a\": rpc error: code = NotFound desc = could not find container \"e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a\": container with ID starting with e802bb824bc129b86f80f3ead287bf5760b4fd81b21aaffdbbb1f1d0bcf2048a not found: ID does not exist"
Sep 09 20:22:36.723434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723407    2582 scope.go:117] "RemoveContainer" containerID="f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"
Sep 09 20:22:36.723639 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:22:36.723623    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e\": container with ID starting with f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e not found: ID does not exist" containerID="f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"
Sep 09 20:22:36.723700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e"} err="failed to get container status \"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e\": rpc error: code = NotFound desc = could not find container \"f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e\": container with ID starting with f024c007df5bbf8561e077d47ce14c7d1fa6ff54a2852d1e03fa587f47a97e2e not found: ID does not exist"
Sep 09 20:22:36.723700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723655    2582 scope.go:117] "RemoveContainer" containerID="fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"
Sep 09 20:22:36.723964 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:22:36.723948    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c\": container with ID starting with fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c not found: ID does not exist" containerID="fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"
Sep 09 20:22:36.724009 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.723967    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c"} err="failed to get container status \"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c\": rpc error: code = NotFound desc = could not find container \"fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c\": container with ID starting with fee90133a5ac439a6ac32fb001ab61005c2fef48473fd6feac30130a15cff11c not found: ID does not exist"
Sep 09 20:22:36.827649 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.827617    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr7qq\" (UniqueName: \"kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq\") pod \"a493e682-6779-4eee-906c-53688381db34\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") "
Sep 09 20:22:36.827779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.827676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config\") pod \"a493e682-6779-4eee-906c-53688381db34\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") "
Sep 09 20:22:36.827779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.827734    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location\") pod \"a493e682-6779-4eee-906c-53688381db34\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") "
Sep 09 20:22:36.827854 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.827794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls\") pod \"a493e682-6779-4eee-906c-53688381db34\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") "
Sep 09 20:22:36.827854 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.827832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert\") pod \"a493e682-6779-4eee-906c-53688381db34\" (UID: \"a493e682-6779-4eee-906c-53688381db34\") "
Sep 09 20:22:36.828138 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.828110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a493e682-6779-4eee-906c-53688381db34" (UID: "a493e682-6779-4eee-906c-53688381db34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:36.828252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.828184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a493e682-6779-4eee-906c-53688381db34" (UID: "a493e682-6779-4eee-906c-53688381db34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:36.828252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.828212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a493e682-6779-4eee-906c-53688381db34" (UID: "a493e682-6779-4eee-906c-53688381db34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:36.829849 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.829819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a493e682-6779-4eee-906c-53688381db34" (UID: "a493e682-6779-4eee-906c-53688381db34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:36.829959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.829945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq" (OuterVolumeSpecName: "kube-api-access-hr7qq") pod "a493e682-6779-4eee-906c-53688381db34" (UID: "a493e682-6779-4eee-906c-53688381db34"). InnerVolumeSpecName "kube-api-access-hr7qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:36.928972 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.928937    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a493e682-6779-4eee-906c-53688381db34-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:22:36.928972 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.928964    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:22:36.928972 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.928974    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr7qq\" (UniqueName: \"kubernetes.io/projected/a493e682-6779-4eee-906c-53688381db34-kube-api-access-hr7qq\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:22:36.929189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.928983    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a493e682-6779-4eee-906c-53688381db34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:22:36.929189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:36.928993    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a493e682-6779-4eee-906c-53688381db34-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:22:37.021223 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:37.021190    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:22:37.023403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:37.023380    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dlq54"]
Sep 09 20:22:37.124562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:22:37.124531    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a493e682-6779-4eee-906c-53688381db34" path="/var/lib/kubelet/pods/a493e682-6779-4eee-906c-53688381db34/volumes"
Sep 09 20:23:02.540663 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.540628    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541009    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kube-rbac-proxy"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541023    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kube-rbac-proxy"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541038    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="storage-initializer"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541043    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="storage-initializer"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kserve-container"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541063    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kserve-container"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kube-rbac-proxy"
Sep 09 20:23:02.541158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.541124    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a493e682-6779-4eee-906c-53688381db34" containerName="kserve-container"
Sep 09 20:23:02.543414 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.543392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.545986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.545963    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 09 20:23:02.545986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.545972    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:23:02.546174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.546060    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:23:02.546174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.546078    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:23:02.546174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.546088    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 09 20:23:02.546174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.546107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:23:02.546174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.546140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:23:02.555167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.555147    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:23:02.644827 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.644791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.644827 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.644829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46t2\" (UniqueName: \"kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.645019 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.644854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.645019 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.644912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.645019 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.644962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746213 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746213 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46t2\" (UniqueName: \"kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.746658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.746639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.747110 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.747090    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.747110 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.747099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.748690 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.748663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.754649 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.754626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46t2\" (UniqueName: \"kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:02.854116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:02.854029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:23:03.127822 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:03.127730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:23:03.130864 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:23:03.130837    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c6c09a_8fcc_44c4_bd7a_9062a36e30ba.slice/crio-a1e911344d4ffb9ce4c73e98e1920cc8b6a6e066cce7c548c30528c64793b8c6 WatchSource:0}: Error finding container a1e911344d4ffb9ce4c73e98e1920cc8b6a6e066cce7c548c30528c64793b8c6: Status 404 returned error can't find the container with id a1e911344d4ffb9ce4c73e98e1920cc8b6a6e066cce7c548c30528c64793b8c6
Sep 09 20:23:03.823674 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:03.823639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerStarted","Data":"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"}
Sep 09 20:23:03.824119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:03.823727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerStarted","Data":"a1e911344d4ffb9ce4c73e98e1920cc8b6a6e066cce7c548c30528c64793b8c6"}
Sep 09 20:23:12.040814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.040777    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:23:12.044146 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.044127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.046966 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.046920    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 09 20:23:12.047154 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.047133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:23:12.058198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.058168    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:23:12.125050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.125020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.125192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.125067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.125192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.125090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.125192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.125175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.125315 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.125242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktj5\" (UniqueName: \"kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226466 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktj5\" (UniqueName: \"kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226918 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:23:12.226780    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 09 20:23:12.226918 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.226878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.226918 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:23:12.226885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls podName:438ba00e-790c-4ad9-a85a-2f2d4f5321aa nodeName:}" failed. No retries permitted until 2026-09-09 20:23:12.726864497 +0000 UTC m=+1682.127541669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 09 20:23:12.227152 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.227131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.227350 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.227329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.258734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.258708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktj5\" (UniqueName: \"kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.730819 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.730782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.733265 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.733244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:12.863725 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.863668    2582 generic.go:356] "Generic (PLEG): container finished" podID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerID="2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3" exitCode=0
Sep 09 20:23:12.863886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.863739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerDied","Data":"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"}
Sep 09 20:23:12.955121 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:12.955091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:13.253982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:13.253901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:23:13.259417 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:23:13.259388    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438ba00e_790c_4ad9_a85a_2f2d4f5321aa.slice/crio-bd7fb983db936ab507343f50dc0d7231dbf81156be3a09f525fdc3d6ebc43e23 WatchSource:0}: Error finding container bd7fb983db936ab507343f50dc0d7231dbf81156be3a09f525fdc3d6ebc43e23: Status 404 returned error can't find the container with id bd7fb983db936ab507343f50dc0d7231dbf81156be3a09f525fdc3d6ebc43e23
Sep 09 20:23:13.872810 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:13.872773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerStarted","Data":"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"}
Sep 09 20:23:13.873001 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:13.872844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerStarted","Data":"bd7fb983db936ab507343f50dc0d7231dbf81156be3a09f525fdc3d6ebc43e23"}
Sep 09 20:23:17.896556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:17.896515    2582 generic.go:356] "Generic (PLEG): container finished" podID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerID="0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478" exitCode=0
Sep 09 20:23:17.896971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:17.896597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerDied","Data":"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"}
Sep 09 20:23:18.905524 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:18.905413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerStarted","Data":"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"}
Sep 09 20:23:18.905524 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:18.905476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerStarted","Data":"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"}
Sep 09 20:23:18.906072 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:18.905714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:18.906072 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:18.905780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:18.932809 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:18.932612    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podStartSLOduration=3.008010878 podStartE2EDuration="6.932593504s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.924582626s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:18.930716482 +0000 UTC m=+1688.331393675" watchObservedRunningTime="2026-09-09 20:23:18.932593504 +0000 UTC m=+1688.333270697"
Sep 09 20:23:24.918134 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:24.918104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:23:54.974628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:23:54.974567    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:24:04.920811 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:04.920766    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:24:14.921277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:14.921240    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:24:22.113964 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:22.113922    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:24:22.114491 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:22.114225    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" containerID="cri-o://7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1" gracePeriod=30
Sep 09 20:24:22.114491 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:22.114261    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" containerID="cri-o://f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6" gracePeriod=30
Sep 09 20:24:22.240404 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:22.240370    2582 generic.go:356] "Generic (PLEG): container finished" podID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerID="f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6" exitCode=2
Sep 09 20:24:22.240554 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:22.240447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerDied","Data":"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"}
Sep 09 20:24:24.912773 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:24.912720    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:29.912783 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:29.912742    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:34.912625 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:34.912581    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:34.913140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:34.912779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:24:34.919335 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:34.919308    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:24:39.912665 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:39.912613    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:44.913299 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:44.913247    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:44.918928 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:44.918888    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:24:49.913189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:49.913144    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:54.912689 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:54.912644    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:24:54.919421 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:54.919385    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:24:54.919589 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:54.919522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:24:59.912955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:24:59.912914    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:25:04.912764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:04.912663    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:25:04.918960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:04.918924    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:25:09.912610 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:09.912571    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:25:14.912886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:14.912838    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 09 20:25:14.919198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:14.919166    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 09 20:25:16.339831 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.339796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:25:16.461715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location\") pod \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") "
Sep 09 20:25:16.461715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert\") pod \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") "
Sep 09 20:25:16.461904 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config\") pod \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") "
Sep 09 20:25:16.461904 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ktj5\" (UniqueName: \"kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5\") pod \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") "
Sep 09 20:25:16.461904 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461857    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") pod \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\" (UID: \"438ba00e-790c-4ad9-a85a-2f2d4f5321aa\") "
Sep 09 20:25:16.462087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.461997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "438ba00e-790c-4ad9-a85a-2f2d4f5321aa" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:16.462154 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.462086    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "438ba00e-790c-4ad9-a85a-2f2d4f5321aa" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:16.462218 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.462167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "438ba00e-790c-4ad9-a85a-2f2d4f5321aa" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:16.463935 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.463907    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "438ba00e-790c-4ad9-a85a-2f2d4f5321aa" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:16.464046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.463994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5" (OuterVolumeSpecName: "kube-api-access-8ktj5") pod "438ba00e-790c-4ad9-a85a-2f2d4f5321aa" (UID: "438ba00e-790c-4ad9-a85a-2f2d4f5321aa"). InnerVolumeSpecName "kube-api-access-8ktj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:16.534957 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.534927    2582 generic.go:356] "Generic (PLEG): container finished" podID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerID="7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1" exitCode=0
Sep 09 20:25:16.535116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.534980    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"
Sep 09 20:25:16.535116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.534989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerDied","Data":"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"}
Sep 09 20:25:16.535116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.535036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl" event={"ID":"438ba00e-790c-4ad9-a85a-2f2d4f5321aa","Type":"ContainerDied","Data":"bd7fb983db936ab507343f50dc0d7231dbf81156be3a09f525fdc3d6ebc43e23"}
Sep 09 20:25:16.535116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.535062    2582 scope.go:117] "RemoveContainer" containerID="f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"
Sep 09 20:25:16.543861 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.543842    2582 scope.go:117] "RemoveContainer" containerID="7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"
Sep 09 20:25:16.552098 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.552069    2582 scope.go:117] "RemoveContainer" containerID="0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"
Sep 09 20:25:16.558424 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.558400    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:25:16.560359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.560339    2582 scope.go:117] "RemoveContainer" containerID="f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"
Sep 09 20:25:16.560659 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:16.560639    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6\": container with ID starting with f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6 not found: ID does not exist" containerID="f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"
Sep 09 20:25:16.560903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.560671    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6"} err="failed to get container status \"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6\": rpc error: code = NotFound desc = could not find container \"f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6\": container with ID starting with f955fd4b4a802d698c7043e9e428f72a10fc3dfc968bba93b83cc428669f80f6 not found: ID does not exist"
Sep 09 20:25:16.560903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.560747    2582 scope.go:117] "RemoveContainer" containerID="7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"
Sep 09 20:25:16.561220 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:16.561194    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1\": container with ID starting with 7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1 not found: ID does not exist" containerID="7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"
Sep 09 20:25:16.561320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.561228    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1"} err="failed to get container status \"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1\": rpc error: code = NotFound desc = could not find container \"7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1\": container with ID starting with 7fb6cd4fb1b12650a15e3677b6543cdc909602e595b3c589d6c544b758284af1 not found: ID does not exist"
Sep 09 20:25:16.561320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.561249    2582 scope.go:117] "RemoveContainer" containerID="0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"
Sep 09 20:25:16.561972 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:16.561782    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478\": container with ID starting with 0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478 not found: ID does not exist" containerID="0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"
Sep 09 20:25:16.561972 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.561818    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478"} err="failed to get container status \"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478\": rpc error: code = NotFound desc = could not find container \"0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478\": container with ID starting with 0bd90e712dc898b674d83688140288e423edefab7b1ace285fb48b65011d8478 not found: ID does not exist"
Sep 09 20:25:16.563407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563350    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:16.563407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563369    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:16.563407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563379    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktj5\" (UniqueName: \"kubernetes.io/projected/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kube-api-access-8ktj5\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:16.563407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563388    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:16.563407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563398    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438ba00e-790c-4ad9-a85a-2f2d4f5321aa-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:16.563724 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:16.563554    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-9x4nl"]
Sep 09 20:25:17.125328 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.125250    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" path="/var/lib/kubelet/pods/438ba00e-790c-4ad9-a85a-2f2d4f5321aa/volumes"
Sep 09 20:25:17.543329 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.543299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerStarted","Data":"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"}
Sep 09 20:25:17.543750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.543356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerStarted","Data":"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"}
Sep 09 20:25:17.543750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.543516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:17.543750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.543661    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:17.544965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.544938    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:25:17.566009 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:17.565958    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podStartSLOduration=2.569573665 podStartE2EDuration="2m15.565942961s" totalImagesPullingTime="2m3.901669661s" totalInitContainerRuntime="9.094699635s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:17.564641555 +0000 UTC m=+1806.965318747" watchObservedRunningTime="2026-09-09 20:25:17.565942961 +0000 UTC m=+1806.966620198"
Sep 09 20:25:18.548225 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:18.548182    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:25:23.552806 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:23.552776    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:23.553313 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:23.553287    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 09 20:25:33.554827 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:33.554796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:47.855805 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:47.855767    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:25:47.856322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:47.856045    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container" containerID="cri-o://8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed" gracePeriod=30
Sep 09 20:25:47.856322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:47.856173    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kube-rbac-proxy" containerID="cri-o://ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4" gracePeriod=30
Sep 09 20:25:48.549358 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:48.549316    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 09 20:25:48.687224 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:48.687192    2582 generic.go:356] "Generic (PLEG): container finished" podID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerID="ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4" exitCode=2
Sep 09 20:25:48.687388 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:48.687248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerDied","Data":"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"}
Sep 09 20:25:52.432050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.432023    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:52.487443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert\") pod \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") "
Sep 09 20:25:52.487587 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g46t2\" (UniqueName: \"kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2\") pod \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") "
Sep 09 20:25:52.487638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls\") pod \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") "
Sep 09 20:25:52.487704 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location\") pod \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") "
Sep 09 20:25:52.487761 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487713    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config\") pod \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\" (UID: \"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba\") "
Sep 09 20:25:52.487826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" (UID: "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:52.487990 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.487971    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:52.488105 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.488076    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" (UID: "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:52.488767 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.488737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" (UID: "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:52.489737 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.489714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" (UID: "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:52.489871 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.489848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2" (OuterVolumeSpecName: "kube-api-access-g46t2") pod "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" (UID: "f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba"). InnerVolumeSpecName "kube-api-access-g46t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:52.588762 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.588663    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46t2\" (UniqueName: \"kubernetes.io/projected/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-api-access-g46t2\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:52.588762 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.588712    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:52.588762 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.588728    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:52.588762 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.588741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:25:52.706854 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.706813    2582 generic.go:356] "Generic (PLEG): container finished" podID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerID="8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed" exitCode=0
Sep 09 20:25:52.707013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.706870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerDied","Data":"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"}
Sep 09 20:25:52.707013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.706879    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"
Sep 09 20:25:52.707013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.706971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9" event={"ID":"f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba","Type":"ContainerDied","Data":"a1e911344d4ffb9ce4c73e98e1920cc8b6a6e066cce7c548c30528c64793b8c6"}
Sep 09 20:25:52.707013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.707010    2582 scope.go:117] "RemoveContainer" containerID="ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"
Sep 09 20:25:52.716182 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.716163    2582 scope.go:117] "RemoveContainer" containerID="8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"
Sep 09 20:25:52.724136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.724114    2582 scope.go:117] "RemoveContainer" containerID="2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"
Sep 09 20:25:52.731392 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.731365    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:25:52.732091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732075    2582 scope.go:117] "RemoveContainer" containerID="ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"
Sep 09 20:25:52.732389 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:52.732369    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4\": container with ID starting with ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4 not found: ID does not exist" containerID="ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"
Sep 09 20:25:52.732469 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732402    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4"} err="failed to get container status \"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4\": rpc error: code = NotFound desc = could not find container \"ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4\": container with ID starting with ce3ae5ad73ba38fbd498e8f21e4a8933310493d879f15bd2b0bb57b62fa30be4 not found: ID does not exist"
Sep 09 20:25:52.732469 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732427    2582 scope.go:117] "RemoveContainer" containerID="8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"
Sep 09 20:25:52.732672 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:52.732650    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed\": container with ID starting with 8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed not found: ID does not exist" containerID="8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"
Sep 09 20:25:52.732748 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732696    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed"} err="failed to get container status \"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed\": rpc error: code = NotFound desc = could not find container \"8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed\": container with ID starting with 8b2567d630431e0f96a15058b7e0fb88b0794423844e1f96ce3f5163097d72ed not found: ID does not exist"
Sep 09 20:25:52.732748 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732718    2582 scope.go:117] "RemoveContainer" containerID="2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"
Sep 09 20:25:52.732961 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:25:52.732943    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3\": container with ID starting with 2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3 not found: ID does not exist" containerID="2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"
Sep 09 20:25:52.733009 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.732966    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3"} err="failed to get container status \"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3\": rpc error: code = NotFound desc = could not find container \"2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3\": container with ID starting with 2c59b4dbfc9a62a1ffc56998c99b7d91cc2c3b6c8114588b127c7db017bbd9f3 not found: ID does not exist"
Sep 09 20:25:52.737625 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:52.737604    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h9vw9"]
Sep 09 20:25:53.086564 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086529    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:25:53.086895 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086882    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086897    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086907    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="storage-initializer"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086913    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="storage-initializer"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086924    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086930    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086942    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container"
Sep 09 20:25:53.086945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086948    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086956    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="storage-initializer"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086961    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="storage-initializer"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086968    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kube-rbac-proxy"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.086974    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kube-rbac-proxy"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.087024    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kserve-container"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.087035    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kube-rbac-proxy"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.087042    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" containerName="kserve-container"
Sep 09 20:25:53.087175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.087049    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="438ba00e-790c-4ad9-a85a-2f2d4f5321aa" containerName="kube-rbac-proxy"
Sep 09 20:25:53.116336 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.116303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:25:53.116505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.116431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.118955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.118931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:25:53.119090 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.118961    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:25:53.119090 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.118969    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:25:53.119090 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.118997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:25:53.119090 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.119014    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:25:53.119090 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.118966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:25:53.119432 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.119332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:25:53.125904 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.125881    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba" path="/var/lib/kubelet/pods/f2c6c09a-8fcc-44c4-bd7a-9062a36e30ba/volumes"
Sep 09 20:25:53.194794 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.194762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hwr\" (UniqueName: \"kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.194794 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.194798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.194968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.194886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.194968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.194916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.194968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.194937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296110 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hwr\" (UniqueName: \"kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296110 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296619 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.296907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.296895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.298738 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.298721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.303074 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.303056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hwr\" (UniqueName: \"kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr\") pod \"isvc-canary-create-predictor-68984b9498-zvwps\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.428446 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.428358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:25:53.718991 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:53.718956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:25:53.721231 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:25:53.721206    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded11f54f_ee0f_483e_a02b_27545a7323a3.slice/crio-c01efc3c8d0a5a5ec0a38391a1216f56ebe88f2cb2f3ec199054f9836f276fb9 WatchSource:0}: Error finding container c01efc3c8d0a5a5ec0a38391a1216f56ebe88f2cb2f3ec199054f9836f276fb9: Status 404 returned error can't find the container with id c01efc3c8d0a5a5ec0a38391a1216f56ebe88f2cb2f3ec199054f9836f276fb9
Sep 09 20:25:54.718937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:54.718901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerStarted","Data":"c68fa3befd7408a582e4ac3146c68554e2e69e3a8ad75a58f976e1a825590226"}
Sep 09 20:25:54.719144 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:54.718969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerStarted","Data":"c01efc3c8d0a5a5ec0a38391a1216f56ebe88f2cb2f3ec199054f9836f276fb9"}
Sep 09 20:25:58.738838 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:58.738618    2582 generic.go:356] "Generic (PLEG): container finished" podID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerID="c68fa3befd7408a582e4ac3146c68554e2e69e3a8ad75a58f976e1a825590226" exitCode=0
Sep 09 20:25:58.738838 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:58.738695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerDied","Data":"c68fa3befd7408a582e4ac3146c68554e2e69e3a8ad75a58f976e1a825590226"}
Sep 09 20:25:58.739949 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:25:58.739929    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:26:12.822727 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:12.822666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerStarted","Data":"468bb7652d814365183a33c6832645d618f0980915bbebd463b2bedbec76aeb7"}
Sep 09 20:26:12.823208 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:12.822760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerStarted","Data":"b7b455d3fb649daefe70a1f055f9acd8842bb96f5cb79f7f08b3913fddb814e6"}
Sep 09 20:26:12.823208 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:12.822989    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:26:12.842796 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:12.842748    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podStartSLOduration=2.519289349 podStartE2EDuration="19.842735398s" totalImagesPullingTime="13.225875129s" totalInitContainerRuntime="4.09757092s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:12.841659258 +0000 UTC m=+1862.242336449" watchObservedRunningTime="2026-09-09 20:26:12.842735398 +0000 UTC m=+1862.243412589"
Sep 09 20:26:13.827300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:13.827255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:26:13.828304 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:13.828279    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:14.831801 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:14.831762    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:19.836162 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:19.836129    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:26:19.836813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:19.836782    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:29.836783 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:29.836745    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:39.836823 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:39.836780    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:49.837427 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:49.837386    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:26:59.837652 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:26:59.837606    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:27:09.837242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:09.837202    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:27:16.872954 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:16.872917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:27:16.873531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:16.873234    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" containerID="cri-o://b7b455d3fb649daefe70a1f055f9acd8842bb96f5cb79f7f08b3913fddb814e6" gracePeriod=30
Sep 09 20:27:16.873531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:16.873274    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" containerID="cri-o://468bb7652d814365183a33c6832645d618f0980915bbebd463b2bedbec76aeb7" gracePeriod=30
Sep 09 20:27:16.873531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:16.873428    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 09 20:27:17.113380 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:17.113345    2582 generic.go:356] "Generic (PLEG): container finished" podID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerID="468bb7652d814365183a33c6832645d618f0980915bbebd463b2bedbec76aeb7" exitCode=2
Sep 09 20:27:17.113536 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:17.113419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerDied","Data":"468bb7652d814365183a33c6832645d618f0980915bbebd463b2bedbec76aeb7"}
Sep 09 20:27:19.832636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:19.832592    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:24.832843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:24.832800    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:26.874645 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:26.874609    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:27:28.226471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.226435    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:27:28.228965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.228940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.231277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.231252    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:27:28.231381 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.231284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:27:28.238451 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.238426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:27:28.300485 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.300452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rpx\" (UniqueName: \"kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.300634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.300496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.300634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.300524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.300634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.300540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.300634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.300571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.306345 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.306321    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:27:28.309000 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.308983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.311293 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.311276    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:27:28.319976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.319950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:27:28.401719 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.401877 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576r4\" (UniqueName: \"kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.401877 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.401877 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.402048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.402048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rpx\" (UniqueName: \"kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.401956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402248 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.402248 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402248 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:27:28.402141    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:27:28.402248 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:27:28.402216    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls podName:5bd90979-aa19-4f78-b170-511bf0041a69 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:28.902193885 +0000 UTC m=+1938.302871056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls") pod "isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:27:28.402666 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.402723 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.402671    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.412956 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.412929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rpx\" (UniqueName: \"kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.503458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.503458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.503458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576r4\" (UniqueName: \"kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.503770 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.503770 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.503903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.503790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.504174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.504154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.504231 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.504214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.505961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.505936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.513190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.513170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-576r4\" (UniqueName: \"kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4\") pod \"isvc-canary-promote-predictor-cb75746f8-kdxfl\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.619384 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.619344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:28.906882 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.906803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:28.907022 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:27:28.906978    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:27:28.907080 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:27:28.907040    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls podName:5bd90979-aa19-4f78-b170-511bf0041a69 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:29.907024356 +0000 UTC m=+1939.307701526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls") pod "isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 20:27:28.921031 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:28.921010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:27:28.923230 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:27:28.923204    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe717493_e303_4cc0_bef1_67f88e3f45f1.slice/crio-7290c947912018ada4596c6ed3bd95ac34c6c0ce42da858c0ebbba68075cda1e WatchSource:0}: Error finding container 7290c947912018ada4596c6ed3bd95ac34c6c0ce42da858c0ebbba68075cda1e: Status 404 returned error can't find the container with id 7290c947912018ada4596c6ed3bd95ac34c6c0ce42da858c0ebbba68075cda1e
Sep 09 20:27:29.181528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.181428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerStarted","Data":"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"}
Sep 09 20:27:29.181528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.181492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerStarted","Data":"7290c947912018ada4596c6ed3bd95ac34c6c0ce42da858c0ebbba68075cda1e"}
Sep 09 20:27:29.832509 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.832469    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:29.832931 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.832576    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:27:29.914646 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.914612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:29.917091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:29.917065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:30.040116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:30.040076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:30.312452 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:30.312420    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:27:30.315920 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:27:30.315896    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd90979_aa19_4f78_b170_511bf0041a69.slice/crio-b2f7acb9f54601bfa3b564a164f1fc185d1a45b868757eea76250e1e3c9e4f1c WatchSource:0}: Error finding container b2f7acb9f54601bfa3b564a164f1fc185d1a45b868757eea76250e1e3c9e4f1c: Status 404 returned error can't find the container with id b2f7acb9f54601bfa3b564a164f1fc185d1a45b868757eea76250e1e3c9e4f1c
Sep 09 20:27:31.198477 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:31.198437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerStarted","Data":"fc2425b3f1a3c9c67967a82f9879080b2db220862cafd87f708dc9cefdb90df2"}
Sep 09 20:27:31.198926 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:31.198522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerStarted","Data":"b2f7acb9f54601bfa3b564a164f1fc185d1a45b868757eea76250e1e3c9e4f1c"}
Sep 09 20:27:33.210734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:33.210700    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerID="b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f" exitCode=0
Sep 09 20:27:33.211114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:33.210780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerDied","Data":"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"}
Sep 09 20:27:34.217401 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.217364    2582 generic.go:356] "Generic (PLEG): container finished" podID="5bd90979-aa19-4f78-b170-511bf0041a69" containerID="fc2425b3f1a3c9c67967a82f9879080b2db220862cafd87f708dc9cefdb90df2" exitCode=0
Sep 09 20:27:34.217813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.217442    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerDied","Data":"fc2425b3f1a3c9c67967a82f9879080b2db220862cafd87f708dc9cefdb90df2"}
Sep 09 20:27:34.219940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.219896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerStarted","Data":"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"}
Sep 09 20:27:34.220091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.219962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerStarted","Data":"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"}
Sep 09 20:27:34.220240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.220226    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:34.220310 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.220271    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:34.221648 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.221621    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:27:34.253596 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.253552    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podStartSLOduration=2.536505986 podStartE2EDuration="6.253537149s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717031163s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:34.251651905 +0000 UTC m=+1943.652329096" watchObservedRunningTime="2026-09-09 20:27:34.253537149 +0000 UTC m=+1943.654214380"
Sep 09 20:27:34.832772 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:34.832731    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:35.226940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:35.226907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerStarted","Data":"9c63b6ef9a4a7387143b652ff8b8f79594ac67ad6f99fe1dc2dcf4ccad16cbf0"}
Sep 09 20:27:35.227348 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:35.226969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerStarted","Data":"b3b3024a46b17043c0b0356376c07333473cfd625b0092c37f94c025e4105710"}
Sep 09 20:27:35.227430 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:35.227373    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:27:35.248733 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:35.248674    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podStartSLOduration=3.451420481 podStartE2EDuration="7.248660018s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797239537s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:35.246570164 +0000 UTC m=+1944.647247359" watchObservedRunningTime="2026-09-09 20:27:35.248660018 +0000 UTC m=+1944.649337209"
Sep 09 20:27:39.832237 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:39.832196    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:40.228046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.228008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:40.228475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.228442    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:40.229441 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.229403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:27:40.232244 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.232224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:27:40.232824 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.232798    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:27:40.232916 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.232829    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:27:40.253139 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:40.253101    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:27:41.257827 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:41.257793    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:27:44.833011 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:44.832966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 09 20:27:47.293319 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.293283    2582 generic.go:356] "Generic (PLEG): container finished" podID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerID="b7b455d3fb649daefe70a1f055f9acd8842bb96f5cb79f7f08b3913fddb814e6" exitCode=137
Sep 09 20:27:47.293692 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.293352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerDied","Data":"b7b455d3fb649daefe70a1f055f9acd8842bb96f5cb79f7f08b3913fddb814e6"}
Sep 09 20:27:47.522645 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.522620    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:27:47.672203 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location\") pod \"ed11f54f-ee0f-483e-a02b-27545a7323a3\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") "
Sep 09 20:27:47.672359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls\") pod \"ed11f54f-ee0f-483e-a02b-27545a7323a3\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") "
Sep 09 20:27:47.672359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672279    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9hwr\" (UniqueName: \"kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr\") pod \"ed11f54f-ee0f-483e-a02b-27545a7323a3\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") "
Sep 09 20:27:47.672359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672303    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert\") pod \"ed11f54f-ee0f-483e-a02b-27545a7323a3\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") "
Sep 09 20:27:47.672524 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672421    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config\") pod \"ed11f54f-ee0f-483e-a02b-27545a7323a3\" (UID: \"ed11f54f-ee0f-483e-a02b-27545a7323a3\") "
Sep 09 20:27:47.672524 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed11f54f-ee0f-483e-a02b-27545a7323a3" (UID: "ed11f54f-ee0f-483e-a02b-27545a7323a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:47.672642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672620    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed11f54f-ee0f-483e-a02b-27545a7323a3" (UID: "ed11f54f-ee0f-483e-a02b-27545a7323a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:47.672886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672864    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:27:47.672991 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672891    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f54f-ee0f-483e-a02b-27545a7323a3-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:27:47.672991 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.672919    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed11f54f-ee0f-483e-a02b-27545a7323a3" (UID: "ed11f54f-ee0f-483e-a02b-27545a7323a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:47.674546 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.674506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr" (OuterVolumeSpecName: "kube-api-access-l9hwr") pod "ed11f54f-ee0f-483e-a02b-27545a7323a3" (UID: "ed11f54f-ee0f-483e-a02b-27545a7323a3"). InnerVolumeSpecName "kube-api-access-l9hwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:47.674987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.674966    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed11f54f-ee0f-483e-a02b-27545a7323a3" (UID: "ed11f54f-ee0f-483e-a02b-27545a7323a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:47.773732 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.773692    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hwr\" (UniqueName: \"kubernetes.io/projected/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-api-access-l9hwr\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:27:47.773732 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.773729    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed11f54f-ee0f-483e-a02b-27545a7323a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:27:47.773965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:47.773743    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed11f54f-ee0f-483e-a02b-27545a7323a3-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:27:48.300644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.300619    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"
Sep 09 20:27:48.301144 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.300619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps" event={"ID":"ed11f54f-ee0f-483e-a02b-27545a7323a3","Type":"ContainerDied","Data":"c01efc3c8d0a5a5ec0a38391a1216f56ebe88f2cb2f3ec199054f9836f276fb9"}
Sep 09 20:27:48.301144 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.300776    2582 scope.go:117] "RemoveContainer" containerID="468bb7652d814365183a33c6832645d618f0980915bbebd463b2bedbec76aeb7"
Sep 09 20:27:48.308888 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.308866    2582 scope.go:117] "RemoveContainer" containerID="b7b455d3fb649daefe70a1f055f9acd8842bb96f5cb79f7f08b3913fddb814e6"
Sep 09 20:27:48.316359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.316343    2582 scope.go:117] "RemoveContainer" containerID="c68fa3befd7408a582e4ac3146c68554e2e69e3a8ad75a58f976e1a825590226"
Sep 09 20:27:48.325774 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.324938    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:27:48.327726 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:48.327702    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-zvwps"]
Sep 09 20:27:49.125647 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:49.125616    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" path="/var/lib/kubelet/pods/ed11f54f-ee0f-483e-a02b-27545a7323a3/volumes"
Sep 09 20:27:50.232852 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:50.232815    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:27:51.258070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:27:51.258029    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:00.233183 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:00.233135    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:28:01.257836 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:01.257794    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:10.233253 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:10.233161    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:28:11.258388 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:11.258348    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:20.232674 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:20.232631    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:28:21.258617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:21.258577    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:30.232899 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:30.232855    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:28:31.258261 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:31.258220    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:40.232866 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:40.232814    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 09 20:28:41.258535 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:41.258491    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:28:46.119886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:46.119855    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:28:49.125044 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:49.125012    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:28:58.240548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:58.240510    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:28:58.241064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:58.240781    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container" containerID="cri-o://9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d" gracePeriod=30
Sep 09 20:28:58.241064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:58.240817    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kube-rbac-proxy" containerID="cri-o://a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b" gracePeriod=30
Sep 09 20:28:58.630546 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:58.630460    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerID="a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b" exitCode=2
Sep 09 20:28:58.630546 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:28:58.630532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerDied","Data":"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"}
Sep 09 20:29:00.228308 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:00.228265    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 09 20:29:02.894317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:02.894294    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:29:03.008810 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.008778    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config\") pod \"fe717493-e303-4cc0-bef1-67f88e3f45f1\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") "
Sep 09 20:29:03.009016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.008819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert\") pod \"fe717493-e303-4cc0-bef1-67f88e3f45f1\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") "
Sep 09 20:29:03.009016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.008842    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location\") pod \"fe717493-e303-4cc0-bef1-67f88e3f45f1\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") "
Sep 09 20:29:03.009016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.008906    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls\") pod \"fe717493-e303-4cc0-bef1-67f88e3f45f1\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") "
Sep 09 20:29:03.009016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.008925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-576r4\" (UniqueName: \"kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4\") pod \"fe717493-e303-4cc0-bef1-67f88e3f45f1\" (UID: \"fe717493-e303-4cc0-bef1-67f88e3f45f1\") "
Sep 09 20:29:03.009236 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.009188    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe717493-e303-4cc0-bef1-67f88e3f45f1" (UID: "fe717493-e303-4cc0-bef1-67f88e3f45f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:03.009236 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.009194    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe717493-e303-4cc0-bef1-67f88e3f45f1" (UID: "fe717493-e303-4cc0-bef1-67f88e3f45f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:03.009313 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.009229    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe717493-e303-4cc0-bef1-67f88e3f45f1" (UID: "fe717493-e303-4cc0-bef1-67f88e3f45f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:03.010936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.010908    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4" (OuterVolumeSpecName: "kube-api-access-576r4") pod "fe717493-e303-4cc0-bef1-67f88e3f45f1" (UID: "fe717493-e303-4cc0-bef1-67f88e3f45f1"). InnerVolumeSpecName "kube-api-access-576r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:03.011079 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.011055    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe717493-e303-4cc0-bef1-67f88e3f45f1" (UID: "fe717493-e303-4cc0-bef1-67f88e3f45f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:03.110140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.110105    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe717493-e303-4cc0-bef1-67f88e3f45f1-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:29:03.110140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.110137    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576r4\" (UniqueName: \"kubernetes.io/projected/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-api-access-576r4\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:29:03.110140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.110146    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:29:03.110380 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.110155    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe717493-e303-4cc0-bef1-67f88e3f45f1-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:29:03.110380 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.110165    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe717493-e303-4cc0-bef1-67f88e3f45f1-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:29:03.653621 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.653588    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerID="9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d" exitCode=0
Sep 09 20:29:03.653813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.653640    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"
Sep 09 20:29:03.653813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.653640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerDied","Data":"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"}
Sep 09 20:29:03.653813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.653783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl" event={"ID":"fe717493-e303-4cc0-bef1-67f88e3f45f1","Type":"ContainerDied","Data":"7290c947912018ada4596c6ed3bd95ac34c6c0ce42da858c0ebbba68075cda1e"}
Sep 09 20:29:03.653927 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.653819    2582 scope.go:117] "RemoveContainer" containerID="a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"
Sep 09 20:29:03.662188 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.662170    2582 scope.go:117] "RemoveContainer" containerID="9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"
Sep 09 20:29:03.669769 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.669748    2582 scope.go:117] "RemoveContainer" containerID="b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"
Sep 09 20:29:03.671435 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.671409    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:29:03.673870 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.673848    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-kdxfl"]
Sep 09 20:29:03.678016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678000    2582 scope.go:117] "RemoveContainer" containerID="a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"
Sep 09 20:29:03.678293 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:29:03.678274    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b\": container with ID starting with a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b not found: ID does not exist" containerID="a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"
Sep 09 20:29:03.678347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678301    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b"} err="failed to get container status \"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b\": rpc error: code = NotFound desc = could not find container \"a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b\": container with ID starting with a6d85387ca5091483b830f7cbcb56c9b6140eb902c1e16ef934149f3568ac16b not found: ID does not exist"
Sep 09 20:29:03.678347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678318    2582 scope.go:117] "RemoveContainer" containerID="9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"
Sep 09 20:29:03.678557 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:29:03.678537    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d\": container with ID starting with 9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d not found: ID does not exist" containerID="9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"
Sep 09 20:29:03.678599 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678566    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d"} err="failed to get container status \"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d\": rpc error: code = NotFound desc = could not find container \"9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d\": container with ID starting with 9ed34b18924b31d08497b1d379fb8b03508b859f93a4bfa401c60d9d4c4eaa4d not found: ID does not exist"
Sep 09 20:29:03.678599 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678586    2582 scope.go:117] "RemoveContainer" containerID="b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"
Sep 09 20:29:03.678837 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:29:03.678821    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f\": container with ID starting with b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f not found: ID does not exist" containerID="b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"
Sep 09 20:29:03.678901 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:03.678840    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f"} err="failed to get container status \"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f\": rpc error: code = NotFound desc = could not find container \"b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f\": container with ID starting with b5e053899dac83bd44b017f7292175891fb174b8c4984b5bef0faca5dc032a4f not found: ID does not exist"
Sep 09 20:29:05.124958 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:29:05.124925    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" path="/var/lib/kubelet/pods/fe717493-e303-4cc0-bef1-67f88e3f45f1/volumes"
Sep 09 20:30:38.512758 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:38.512723    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:30:38.513242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:38.512961    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" containerID="cri-o://b3b3024a46b17043c0b0356376c07333473cfd625b0092c37f94c025e4105710" gracePeriod=30
Sep 09 20:30:38.513242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:38.512984    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kube-rbac-proxy" containerID="cri-o://9c63b6ef9a4a7387143b652ff8b8f79594ac67ad6f99fe1dc2dcf4ccad16cbf0" gracePeriod=30
Sep 09 20:30:39.083368 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:39.083334    2582 generic.go:356] "Generic (PLEG): container finished" podID="5bd90979-aa19-4f78-b170-511bf0041a69" containerID="9c63b6ef9a4a7387143b652ff8b8f79594ac67ad6f99fe1dc2dcf4ccad16cbf0" exitCode=2
Sep 09 20:30:39.083534 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:39.083407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerDied","Data":"9c63b6ef9a4a7387143b652ff8b8f79594ac67ad6f99fe1dc2dcf4ccad16cbf0"}
Sep 09 20:30:39.119911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:39.119874    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 09 20:30:40.233639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:40.233596    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 09 20:30:43.103734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.103668    2582 generic.go:356] "Generic (PLEG): container finished" podID="5bd90979-aa19-4f78-b170-511bf0041a69" containerID="b3b3024a46b17043c0b0356376c07333473cfd625b0092c37f94c025e4105710" exitCode=0
Sep 09 20:30:43.103734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.103717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerDied","Data":"b3b3024a46b17043c0b0356376c07333473cfd625b0092c37f94c025e4105710"}
Sep 09 20:30:43.155574 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.155552    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:30:43.249092 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249010    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config\") pod \"5bd90979-aa19-4f78-b170-511bf0041a69\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") "
Sep 09 20:30:43.249092 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location\") pod \"5bd90979-aa19-4f78-b170-511bf0041a69\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") "
Sep 09 20:30:43.249276 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249104    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert\") pod \"5bd90979-aa19-4f78-b170-511bf0041a69\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") "
Sep 09 20:30:43.249384 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249361    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22rpx\" (UniqueName: \"kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx\") pod \"5bd90979-aa19-4f78-b170-511bf0041a69\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") "
Sep 09 20:30:43.249442 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249405    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") pod \"5bd90979-aa19-4f78-b170-511bf0041a69\" (UID: \"5bd90979-aa19-4f78-b170-511bf0041a69\") "
Sep 09 20:30:43.249442 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bd90979-aa19-4f78-b170-511bf0041a69" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:43.249555 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249439    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bd90979-aa19-4f78-b170-511bf0041a69" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:43.249555 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249465    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bd90979-aa19-4f78-b170-511bf0041a69" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:43.249814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249792    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd90979-aa19-4f78-b170-511bf0041a69-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:30:43.249814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249814    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:30:43.250021 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.249828    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bd90979-aa19-4f78-b170-511bf0041a69-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:30:43.252010 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.251980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx" (OuterVolumeSpecName: "kube-api-access-22rpx") pod "5bd90979-aa19-4f78-b170-511bf0041a69" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69"). InnerVolumeSpecName "kube-api-access-22rpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:43.252010 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.251982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bd90979-aa19-4f78-b170-511bf0041a69" (UID: "5bd90979-aa19-4f78-b170-511bf0041a69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:43.350363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.350332    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rpx\" (UniqueName: \"kubernetes.io/projected/5bd90979-aa19-4f78-b170-511bf0041a69-kube-api-access-22rpx\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:30:43.350363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.350361    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bd90979-aa19-4f78-b170-511bf0041a69-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:30:43.655298 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655219    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:30:43.655599 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655586    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kube-rbac-proxy"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655613    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kube-rbac-proxy"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655621    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="storage-initializer"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655627    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="storage-initializer"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655634    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="storage-initializer"
Sep 09 20:30:43.655642 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655640    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="storage-initializer"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655647    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655653    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655661    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655721    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655737    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655756    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655762    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655769    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="storage-initializer"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655775    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="storage-initializer"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655839    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655854    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655864    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" containerName="kube-rbac-proxy"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655873    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655886    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed11f54f-ee0f-483e-a02b-27545a7323a3" containerName="kserve-container"
Sep 09 20:30:43.656058 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.655893    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe717493-e303-4cc0-bef1-67f88e3f45f1" containerName="kube-rbac-proxy"
Sep 09 20:30:43.659335 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.659319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.661573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.661548    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 20:30:43.661675 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.661589    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:30:43.665986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.665962    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:30:43.669944 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.669922    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:30:43.670034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.669971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.672064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.672045    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:30:43.682355 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.682332    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:30:43.754429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.754401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmstg\" (UniqueName: \"kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.754578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.754443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.754578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.754463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.754578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.754550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.754726 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.754583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855053 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855234 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.855234 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.855234 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855387 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhszr\" (UniqueName: \"kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.855451 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.855506 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.855506 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmstg\" (UniqueName: \"kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.855617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.855573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.856157 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.856135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.856346 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.856322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.857730 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.857704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.862814 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.862795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmstg\" (UniqueName: \"kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-g87r9\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:43.956230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhszr\" (UniqueName: \"kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956436 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:30:43.956366    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:30:43.956755 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:30:43.956454    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls podName:768ebc47-dc61-4ad1-a19d-eafef98dd46b nodeName:}" failed. No retries permitted until 2026-09-09 20:30:44.456435418 +0000 UTC m=+2133.857112588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:30:43.956831 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.956990 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.956970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.957069 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.957036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.966540 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.966516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhszr\" (UniqueName: \"kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:43.971382 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:43.971364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:44.111534 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.111510    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"
Sep 09 20:30:44.111908 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.111509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq" event={"ID":"5bd90979-aa19-4f78-b170-511bf0041a69","Type":"ContainerDied","Data":"b2f7acb9f54601bfa3b564a164f1fc185d1a45b868757eea76250e1e3c9e4f1c"}
Sep 09 20:30:44.111908 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.111671    2582 scope.go:117] "RemoveContainer" containerID="9c63b6ef9a4a7387143b652ff8b8f79594ac67ad6f99fe1dc2dcf4ccad16cbf0"
Sep 09 20:30:44.122574 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.122553    2582 scope.go:117] "RemoveContainer" containerID="b3b3024a46b17043c0b0356376c07333473cfd625b0092c37f94c025e4105710"
Sep 09 20:30:44.130020 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.130003    2582 scope.go:117] "RemoveContainer" containerID="fc2425b3f1a3c9c67967a82f9879080b2db220862cafd87f708dc9cefdb90df2"
Sep 09 20:30:44.136998 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.136971    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:30:44.138490 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.138471    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-bkjtq"]
Sep 09 20:30:44.274695 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.274660    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:30:44.276904 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:30:44.276879    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29d0d51_aa53_4365_b193_6d8d50e9d1fb.slice/crio-b7360e07a5aacd43ec2e65e026560a0a74d906035253c40801244a15b99c5ae3 WatchSource:0}: Error finding container b7360e07a5aacd43ec2e65e026560a0a74d906035253c40801244a15b99c5ae3: Status 404 returned error can't find the container with id b7360e07a5aacd43ec2e65e026560a0a74d906035253c40801244a15b99c5ae3
Sep 09 20:30:44.460789 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.460749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:44.463216 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.463184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-jpnpw\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:44.582219 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.582138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:44.888615 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:44.888591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:30:44.891888 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:30:44.891865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768ebc47_dc61_4ad1_a19d_eafef98dd46b.slice/crio-1ad387737cef4b1301ab87c386b6f2f5601f97b51bb929da59a3d8cfcf5c5701 WatchSource:0}: Error finding container 1ad387737cef4b1301ab87c386b6f2f5601f97b51bb929da59a3d8cfcf5c5701: Status 404 returned error can't find the container with id 1ad387737cef4b1301ab87c386b6f2f5601f97b51bb929da59a3d8cfcf5c5701
Sep 09 20:30:45.118016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:45.117977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerStarted","Data":"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"}
Sep 09 20:30:45.118441 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:45.118050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerStarted","Data":"1ad387737cef4b1301ab87c386b6f2f5601f97b51bb929da59a3d8cfcf5c5701"}
Sep 09 20:30:45.124969 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:45.124940    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd90979-aa19-4f78-b170-511bf0041a69" path="/var/lib/kubelet/pods/5bd90979-aa19-4f78-b170-511bf0041a69/volumes"
Sep 09 20:30:45.125341 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:45.125320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerStarted","Data":"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"}
Sep 09 20:30:45.125433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:45.125382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerStarted","Data":"b7360e07a5aacd43ec2e65e026560a0a74d906035253c40801244a15b99c5ae3"}
Sep 09 20:30:48.140511 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:48.140436    2582 generic.go:356] "Generic (PLEG): container finished" podID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerID="217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726" exitCode=0
Sep 09 20:30:48.140511 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:48.140493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerDied","Data":"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"}
Sep 09 20:30:49.146594 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.146556    2582 generic.go:356] "Generic (PLEG): container finished" podID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerID="7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12" exitCode=0
Sep 09 20:30:49.147018 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.146628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerDied","Data":"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"}
Sep 09 20:30:49.149038 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.149018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerStarted","Data":"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"}
Sep 09 20:30:49.149163 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.149062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerStarted","Data":"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"}
Sep 09 20:30:49.149252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.149234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:49.149310 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.149284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:49.150595 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.150569    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:30:49.184299 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:49.184256    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podStartSLOduration=2.462339353 podStartE2EDuration="6.184241306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.721901953s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:49.18205436 +0000 UTC m=+2138.582731558" watchObservedRunningTime="2026-09-09 20:30:49.184241306 +0000 UTC m=+2138.584918497"
Sep 09 20:30:50.157733 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.157696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerStarted","Data":"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"}
Sep 09 20:30:50.158145 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.157772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerStarted","Data":"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"}
Sep 09 20:30:50.158145 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.157955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:30:50.158274 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.158209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:50.158274 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.158267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:50.159695 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.159658    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:30:50.178033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:50.177990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podStartSLOduration=3.171367166 podStartE2EDuration="7.177976446s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00660928s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:50.177240376 +0000 UTC m=+2139.577917572" watchObservedRunningTime="2026-09-09 20:30:50.177976446 +0000 UTC m=+2139.578653686"
Sep 09 20:30:51.161553 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:51.161511    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:30:55.162226 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:55.162194    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:30:55.162795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:55.162767    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:30:56.165635 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:56.165600    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:30:56.166279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:30:56.166248    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:05.163186 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:05.163098    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:06.166807 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:06.166761    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:15.163044 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:15.163003    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:16.166289 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:16.166249    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:25.162999 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:25.162961    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:26.166170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:26.166125    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:35.162935 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:35.162889    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:36.166846 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:36.166804    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:45.163517 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:45.163475    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:46.167048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:46.167008    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:31:55.163057 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:55.163018    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 09 20:31:56.166807 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:31:56.166771    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 09 20:32:02.119617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.119579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:32:02.252528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.252491    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:32:02.252772 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.252745    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container" containerID="cri-o://6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f" gracePeriod=30
Sep 09 20:32:02.252866 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.252796    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kube-rbac-proxy" containerID="cri-o://0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981" gracePeriod=30
Sep 09 20:32:02.253568 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.253548    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:32:02.495544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.495511    2582 generic.go:356] "Generic (PLEG): container finished" podID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerID="0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981" exitCode=2
Sep 09 20:32:02.495727 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:02.495583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerDied","Data":"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"}
Sep 09 20:32:06.162128 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.162087    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 09 20:32:06.899929 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.899905    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:32:06.984305 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert\") pod \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") "
Sep 09 20:32:06.984458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984328    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") pod \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") "
Sep 09 20:32:06.984458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location\") pod \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") "
Sep 09 20:32:06.984458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984402    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhszr\" (UniqueName: \"kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr\") pod \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") "
Sep 09 20:32:06.984458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config\") pod \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\" (UID: \"768ebc47-dc61-4ad1-a19d-eafef98dd46b\") "
Sep 09 20:32:06.984719 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "768ebc47-dc61-4ad1-a19d-eafef98dd46b" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:06.984859 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984781    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "768ebc47-dc61-4ad1-a19d-eafef98dd46b" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:06.984927 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.984882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "768ebc47-dc61-4ad1-a19d-eafef98dd46b" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:06.986388 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.986366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr" (OuterVolumeSpecName: "kube-api-access-hhszr") pod "768ebc47-dc61-4ad1-a19d-eafef98dd46b" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b"). InnerVolumeSpecName "kube-api-access-hhszr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:06.986445 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:06.986385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "768ebc47-dc61-4ad1-a19d-eafef98dd46b" (UID: "768ebc47-dc61-4ad1-a19d-eafef98dd46b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:07.086126 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.086076    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:07.086126 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.086122    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhszr\" (UniqueName: \"kubernetes.io/projected/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-api-access-hhszr\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:07.086126 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.086132    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:07.086126 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.086142    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/768ebc47-dc61-4ad1-a19d-eafef98dd46b-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:07.086387 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.086152    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768ebc47-dc61-4ad1-a19d-eafef98dd46b-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:07.519663 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.519626    2582 generic.go:356] "Generic (PLEG): container finished" podID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerID="6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f" exitCode=0
Sep 09 20:32:07.520087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.519714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerDied","Data":"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"}
Sep 09 20:32:07.520087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.519737    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"
Sep 09 20:32:07.520087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.519778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw" event={"ID":"768ebc47-dc61-4ad1-a19d-eafef98dd46b","Type":"ContainerDied","Data":"1ad387737cef4b1301ab87c386b6f2f5601f97b51bb929da59a3d8cfcf5c5701"}
Sep 09 20:32:07.520087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.519806    2582 scope.go:117] "RemoveContainer" containerID="0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"
Sep 09 20:32:07.528415 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.528399    2582 scope.go:117] "RemoveContainer" containerID="6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"
Sep 09 20:32:07.535633 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.535615    2582 scope.go:117] "RemoveContainer" containerID="7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"
Sep 09 20:32:07.538046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.538023    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:32:07.544001 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.543979    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-jpnpw"]
Sep 09 20:32:07.544285 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.544254    2582 scope.go:117] "RemoveContainer" containerID="0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"
Sep 09 20:32:07.544548 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:07.544528    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981\": container with ID starting with 0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981 not found: ID does not exist" containerID="0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"
Sep 09 20:32:07.544621 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.544556    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981"} err="failed to get container status \"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981\": rpc error: code = NotFound desc = could not find container \"0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981\": container with ID starting with 0d4945b1a751159c296cf601888d00bb2060c68ff7f359d6dc0c0bdb5a249981 not found: ID does not exist"
Sep 09 20:32:07.544621 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.544574    2582 scope.go:117] "RemoveContainer" containerID="6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"
Sep 09 20:32:07.544839 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:07.544824    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f\": container with ID starting with 6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f not found: ID does not exist" containerID="6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"
Sep 09 20:32:07.544881 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.544844    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f"} err="failed to get container status \"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f\": rpc error: code = NotFound desc = could not find container \"6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f\": container with ID starting with 6944836ca432f32f0c4e5a2aebe6ec9a0fee3b280e7a24faef6382f4a572d91f not found: ID does not exist"
Sep 09 20:32:07.544881 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.544856    2582 scope.go:117] "RemoveContainer" containerID="7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"
Sep 09 20:32:07.545064 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:07.545049    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12\": container with ID starting with 7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12 not found: ID does not exist" containerID="7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"
Sep 09 20:32:07.545107 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:07.545067    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12"} err="failed to get container status \"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12\": rpc error: code = NotFound desc = could not find container \"7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12\": container with ID starting with 7693f4fb67d2f29b2216d9ab80bc4df98063eae073b1bcbce0629291020afa12 not found: ID does not exist"
Sep 09 20:32:09.130277 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:09.130238    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" path="/var/lib/kubelet/pods/768ebc47-dc61-4ad1-a19d-eafef98dd46b/volumes"
Sep 09 20:32:23.619778 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.619743    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:32:23.620664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.620626    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container" containerID="cri-o://52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3" gracePeriod=30
Sep 09 20:32:23.620990 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.620926    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kube-rbac-proxy" containerID="cri-o://be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab" gracePeriod=30
Sep 09 20:32:23.725240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725206    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:32:23.725621 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container"
Sep 09 20:32:23.725672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725623    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container"
Sep 09 20:32:23.725672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725634    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kube-rbac-proxy"
Sep 09 20:32:23.725672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725640    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kube-rbac-proxy"
Sep 09 20:32:23.725672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725650    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="storage-initializer"
Sep 09 20:32:23.725672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725657    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="storage-initializer"
Sep 09 20:32:23.725903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725740    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kube-rbac-proxy"
Sep 09 20:32:23.725903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.725753    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="768ebc47-dc61-4ad1-a19d-eafef98dd46b" containerName="kserve-container"
Sep 09 20:32:23.728731 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.728707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:23.731186 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.731141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:32:23.740715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.740690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:32:23.921038 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.920953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:23.921038 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.921003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:23.921223 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.921048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:23.921223 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.921112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrl7l\" (UniqueName: \"kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:23.921223 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:23.921131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022038 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022214 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrl7l\" (UniqueName: \"kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022214 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022214 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022214 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022864 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.022923 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.022840    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.024459 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.024440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.030258 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.030235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrl7l\" (UniqueName: \"kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-v4ns8\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.041064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.041040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:24.343746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.343665    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:32:24.346117 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:32:24.346087    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca34996c_0b01_413a_bfbf_c5629211f4ef.slice/crio-da300988c6440b4eaee9e371d99acc4c08564489f1e4b2b36f7cff963b235863 WatchSource:0}: Error finding container da300988c6440b4eaee9e371d99acc4c08564489f1e4b2b36f7cff963b235863: Status 404 returned error can't find the container with id da300988c6440b4eaee9e371d99acc4c08564489f1e4b2b36f7cff963b235863
Sep 09 20:32:24.596939 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.596836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerStarted","Data":"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"}
Sep 09 20:32:24.596939 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.596903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerStarted","Data":"da300988c6440b4eaee9e371d99acc4c08564489f1e4b2b36f7cff963b235863"}
Sep 09 20:32:24.599123 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.599096    2582 generic.go:356] "Generic (PLEG): container finished" podID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerID="be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab" exitCode=2
Sep 09 20:32:24.599279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:24.599168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerDied","Data":"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"}
Sep 09 20:32:25.159004 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:25.158960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 09 20:32:28.168960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.168936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:32:28.255151 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location\") pod \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") "
Sep 09 20:32:28.255151 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert\") pod \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") "
Sep 09 20:32:28.255366 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255175    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmstg\" (UniqueName: \"kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg\") pod \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") "
Sep 09 20:32:28.255366 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config\") pod \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") "
Sep 09 20:32:28.255366 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255257    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls\") pod \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\" (UID: \"d29d0d51-aa53-4365-b193-6d8d50e9d1fb\") "
Sep 09 20:32:28.255528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255379    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d29d0d51-aa53-4365-b193-6d8d50e9d1fb" (UID: "d29d0d51-aa53-4365-b193-6d8d50e9d1fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:28.255528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255517    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d29d0d51-aa53-4365-b193-6d8d50e9d1fb" (UID: "d29d0d51-aa53-4365-b193-6d8d50e9d1fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:28.255638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255560    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d29d0d51-aa53-4365-b193-6d8d50e9d1fb" (UID: "d29d0d51-aa53-4365-b193-6d8d50e9d1fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:28.255638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255627    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:28.255638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255638    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:28.255771 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.255648    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:28.257379 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.257355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg" (OuterVolumeSpecName: "kube-api-access-tmstg") pod "d29d0d51-aa53-4365-b193-6d8d50e9d1fb" (UID: "d29d0d51-aa53-4365-b193-6d8d50e9d1fb"). InnerVolumeSpecName "kube-api-access-tmstg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:28.257462 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.257361    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d29d0d51-aa53-4365-b193-6d8d50e9d1fb" (UID: "d29d0d51-aa53-4365-b193-6d8d50e9d1fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:28.356617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.356586    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:28.356617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.356616    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmstg\" (UniqueName: \"kubernetes.io/projected/d29d0d51-aa53-4365-b193-6d8d50e9d1fb-kube-api-access-tmstg\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:32:28.619478 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.619389    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerID="6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540" exitCode=0
Sep 09 20:32:28.619643 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.619466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerDied","Data":"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"}
Sep 09 20:32:28.621420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.621397    2582 generic.go:356] "Generic (PLEG): container finished" podID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerID="52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3" exitCode=0
Sep 09 20:32:28.621548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.621430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerDied","Data":"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"}
Sep 09 20:32:28.621548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.621455    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"
Sep 09 20:32:28.621548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.621472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9" event={"ID":"d29d0d51-aa53-4365-b193-6d8d50e9d1fb","Type":"ContainerDied","Data":"b7360e07a5aacd43ec2e65e026560a0a74d906035253c40801244a15b99c5ae3"}
Sep 09 20:32:28.621548 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.621498    2582 scope.go:117] "RemoveContainer" containerID="be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"
Sep 09 20:32:28.632906 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.632888    2582 scope.go:117] "RemoveContainer" containerID="52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"
Sep 09 20:32:28.644401 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.644376    2582 scope.go:117] "RemoveContainer" containerID="217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"
Sep 09 20:32:28.650167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.650143    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:32:28.654064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.654042    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-g87r9"]
Sep 09 20:32:28.662307 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.662287    2582 scope.go:117] "RemoveContainer" containerID="be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"
Sep 09 20:32:28.662636 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:28.662618    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab\": container with ID starting with be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab not found: ID does not exist" containerID="be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"
Sep 09 20:32:28.662723 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.662643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab"} err="failed to get container status \"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab\": rpc error: code = NotFound desc = could not find container \"be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab\": container with ID starting with be170c96352f1c3605a85bd4dc89d09c0692555df92807a6aab78f86b5f3d6ab not found: ID does not exist"
Sep 09 20:32:28.662723 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.662662    2582 scope.go:117] "RemoveContainer" containerID="52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"
Sep 09 20:32:28.662981 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:28.662948    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3\": container with ID starting with 52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3 not found: ID does not exist" containerID="52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"
Sep 09 20:32:28.663037 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.662987    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3"} err="failed to get container status \"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3\": rpc error: code = NotFound desc = could not find container \"52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3\": container with ID starting with 52d3d83d7d6aedfb579553e490ce6bc516afcf45b0a218932a01ccc279fa04b3 not found: ID does not exist"
Sep 09 20:32:28.663037 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.663005    2582 scope.go:117] "RemoveContainer" containerID="217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"
Sep 09 20:32:28.663281 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:32:28.663253    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726\": container with ID starting with 217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726 not found: ID does not exist" containerID="217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"
Sep 09 20:32:28.663368 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:28.663289    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726"} err="failed to get container status \"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726\": rpc error: code = NotFound desc = could not find container \"217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726\": container with ID starting with 217f8afeddc51597d7fc70da19da17f23c60cdee86ccc2bc97bdc421200dd726 not found: ID does not exist"
Sep 09 20:32:29.125106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.125071    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" path="/var/lib/kubelet/pods/d29d0d51-aa53-4365-b193-6d8d50e9d1fb/volumes"
Sep 09 20:32:29.628170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.628134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerStarted","Data":"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"}
Sep 09 20:32:29.628607 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.628196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerStarted","Data":"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"}
Sep 09 20:32:29.628607 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.628441    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:29.628607 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.628589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:29.629728 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.629704    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:32:29.648747 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:29.648704    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podStartSLOduration=2.73224011 podStartE2EDuration="6.648662761s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.916422651s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:29.648519154 +0000 UTC m=+2239.049196351" watchObservedRunningTime="2026-09-09 20:32:29.648662761 +0000 UTC m=+2239.049339965"
Sep 09 20:32:30.633874 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:30.633836    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:32:35.638858 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:35.638780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:32:35.639459 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:35.639429    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:32:45.640366 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:45.640325    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:32:51.631853 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.631819    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:32:51.632221 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632181    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kube-rbac-proxy"
Sep 09 20:32:51.632221 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632193    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kube-rbac-proxy"
Sep 09 20:32:51.632221 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632214    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container"
Sep 09 20:32:51.632326 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632222    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container"
Sep 09 20:32:51.632326 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632237    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="storage-initializer"
Sep 09 20:32:51.632326 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632243    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="storage-initializer"
Sep 09 20:32:51.632326 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632323    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kube-rbac-proxy"
Sep 09 20:32:51.632444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.632331    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29d0d51-aa53-4365-b193-6d8d50e9d1fb" containerName="kserve-container"
Sep 09 20:32:51.635490 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.635474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.638265 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.638245    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 20:32:51.638403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.638319    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:32:51.646299 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.646277    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:32:51.754770 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.754738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.754910 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.754785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.754910 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.754894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.754997 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.754935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnf9\" (UniqueName: \"kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.754997 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.754956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856302 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856540 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856540 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856540 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnf9\" (UniqueName: \"kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856540 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.856923 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.856895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.857097 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.857077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.857165 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.857098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.858897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.858877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.867510 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.867481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnf9\" (UniqueName: \"kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:51.947333 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:51.947252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:52.221167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:52.221141    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:32:52.223099 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:32:52.223073    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c01275_441f_4afb_a7d2_6c6fd7c0b50b.slice/crio-162ed09179309059290e89ac11b82fb7f3400808e42a0c40f5189acb6ca7df5d WatchSource:0}: Error finding container 162ed09179309059290e89ac11b82fb7f3400808e42a0c40f5189acb6ca7df5d: Status 404 returned error can't find the container with id 162ed09179309059290e89ac11b82fb7f3400808e42a0c40f5189acb6ca7df5d
Sep 09 20:32:52.740300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:52.740262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerStarted","Data":"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"}
Sep 09 20:32:52.740693 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:52.740331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerStarted","Data":"162ed09179309059290e89ac11b82fb7f3400808e42a0c40f5189acb6ca7df5d"}
Sep 09 20:32:55.639330 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:55.639291    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:32:56.760147 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:56.760113    2582 generic.go:356] "Generic (PLEG): container finished" podID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerID="b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856" exitCode=0
Sep 09 20:32:56.760527 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:56.760162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerDied","Data":"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"}
Sep 09 20:32:57.766546 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:57.766509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerStarted","Data":"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"}
Sep 09 20:32:57.767007 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:57.766575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerStarted","Data":"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"}
Sep 09 20:32:57.767007 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:57.766842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:32:57.786591 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:57.786541    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" podStartSLOduration=2.9816920209999997 podStartE2EDuration="6.786527221s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8048352s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:57.78550833 +0000 UTC m=+2267.186185522" watchObservedRunningTime="2026-09-09 20:32:57.786527221 +0000 UTC m=+2267.187204413"
Sep 09 20:32:58.771343 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:32:58.771314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:33:04.781092 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:04.781059    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:33:05.640197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:05.640157    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:33:15.639987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:15.639944    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:33:25.640219 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:25.640175    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:33:34.785715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:34.785669    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:33:35.639356 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:35.639318    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:33:41.693529 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:41.693494    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:33:41.694032 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:41.693760    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kserve-container" containerID="cri-o://090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff" gracePeriod=30
Sep 09 20:33:41.694429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:41.694019    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kube-rbac-proxy" containerID="cri-o://5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72" gracePeriod=30
Sep 09 20:33:41.968508 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:41.968476    2582 generic.go:356] "Generic (PLEG): container finished" podID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerID="5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72" exitCode=2
Sep 09 20:33:41.968710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:41.968552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerDied","Data":"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"}
Sep 09 20:33:44.776927 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:44.776873    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 09 20:33:44.781736 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:44.781710    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 09 20:33:45.639837 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:45.639805    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:33:46.974414 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:46.974382    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:33:46.978444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:46.978422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:46.981287 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:46.981263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 20:33:46.981390 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:46.981288    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:33:46.991189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:46.991165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:33:47.015181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.015150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc6g\" (UniqueName: \"kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.015311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.015204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.015311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.015274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.015311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.015300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.015422 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.015357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116015 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.115980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc6g\" (UniqueName: \"kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116181 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116649 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.116861 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.117018 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.116995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.118635 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.118614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.131974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.131943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc6g\" (UniqueName: \"kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-jxslk\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.290846 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.290759    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:47.563911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.563889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:33:47.566323 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:33:47.566292    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7960bebc_b48c_4f12_a2dc_48acf55fbe47.slice/crio-aef2554d1d9d7a5edb48d28f232495a51583143ae08603b6e552f2cfb8263be4 WatchSource:0}: Error finding container aef2554d1d9d7a5edb48d28f232495a51583143ae08603b6e552f2cfb8263be4: Status 404 returned error can't find the container with id aef2554d1d9d7a5edb48d28f232495a51583143ae08603b6e552f2cfb8263be4
Sep 09 20:33:47.998560 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.998522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerStarted","Data":"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"}
Sep 09 20:33:47.998999 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:47.998597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerStarted","Data":"aef2554d1d9d7a5edb48d28f232495a51583143ae08603b6e552f2cfb8263be4"}
Sep 09 20:33:48.142454 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.142432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:33:48.226793 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.226758    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config\") pod \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") "
Sep 09 20:33:48.226793 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.226790    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbnf9\" (UniqueName: \"kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9\") pod \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") "
Sep 09 20:33:48.227034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.226828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls\") pod \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") "
Sep 09 20:33:48.227034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.226851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location\") pod \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") "
Sep 09 20:33:48.227034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.226872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert\") pod \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\" (UID: \"89c01275-441f-4afb-a7d2-6c6fd7c0b50b\") "
Sep 09 20:33:48.227207 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.227161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89c01275-441f-4afb-a7d2-6c6fd7c0b50b" (UID: "89c01275-441f-4afb-a7d2-6c6fd7c0b50b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:48.227279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.227130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89c01275-441f-4afb-a7d2-6c6fd7c0b50b" (UID: "89c01275-441f-4afb-a7d2-6c6fd7c0b50b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:48.227337 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.227300    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89c01275-441f-4afb-a7d2-6c6fd7c0b50b" (UID: "89c01275-441f-4afb-a7d2-6c6fd7c0b50b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:48.229023 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.228999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9" (OuterVolumeSpecName: "kube-api-access-nbnf9") pod "89c01275-441f-4afb-a7d2-6c6fd7c0b50b" (UID: "89c01275-441f-4afb-a7d2-6c6fd7c0b50b"). InnerVolumeSpecName "kube-api-access-nbnf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:48.229124 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.229044    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89c01275-441f-4afb-a7d2-6c6fd7c0b50b" (UID: "89c01275-441f-4afb-a7d2-6c6fd7c0b50b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:48.327710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.327638    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:33:48.327710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.327695    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnf9\" (UniqueName: \"kubernetes.io/projected/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kube-api-access-nbnf9\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:33:48.327710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.327712    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:33:48.327710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.327725    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:33:48.327968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:48.327737    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89c01275-441f-4afb-a7d2-6c6fd7c0b50b-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:33:49.004564 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.004527    2582 generic.go:356] "Generic (PLEG): container finished" podID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerID="090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff" exitCode=0
Sep 09 20:33:49.005013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.004584    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"
Sep 09 20:33:49.005013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.004615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerDied","Data":"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"}
Sep 09 20:33:49.005013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.004721    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp" event={"ID":"89c01275-441f-4afb-a7d2-6c6fd7c0b50b","Type":"ContainerDied","Data":"162ed09179309059290e89ac11b82fb7f3400808e42a0c40f5189acb6ca7df5d"}
Sep 09 20:33:49.005013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.004761    2582 scope.go:117] "RemoveContainer" containerID="5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"
Sep 09 20:33:49.013571 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.013554    2582 scope.go:117] "RemoveContainer" containerID="090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"
Sep 09 20:33:49.020821 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.020804    2582 scope.go:117] "RemoveContainer" containerID="b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"
Sep 09 20:33:49.027244 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.027220    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:33:49.028734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.028721    2582 scope.go:117] "RemoveContainer" containerID="5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"
Sep 09 20:33:49.029001 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:33:49.028983    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72\": container with ID starting with 5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72 not found: ID does not exist" containerID="5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"
Sep 09 20:33:49.029060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.029007    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72"} err="failed to get container status \"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72\": rpc error: code = NotFound desc = could not find container \"5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72\": container with ID starting with 5b0b119fd7050ac70724bc1dbd8f9ffb0ad86525b2c047dddf5077b16ca80a72 not found: ID does not exist"
Sep 09 20:33:49.029060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.029023    2582 scope.go:117] "RemoveContainer" containerID="090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"
Sep 09 20:33:49.029272 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:33:49.029252    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff\": container with ID starting with 090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff not found: ID does not exist" containerID="090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"
Sep 09 20:33:49.029317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.029279    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff"} err="failed to get container status \"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff\": rpc error: code = NotFound desc = could not find container \"090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff\": container with ID starting with 090017afe63ec162a8d646ea896cde0a378f2df3b9aab86e0bbfefa06bfd79ff not found: ID does not exist"
Sep 09 20:33:49.029317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.029295    2582 scope.go:117] "RemoveContainer" containerID="b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"
Sep 09 20:33:49.029556 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:33:49.029535    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856\": container with ID starting with b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856 not found: ID does not exist" containerID="b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"
Sep 09 20:33:49.029639 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.029559    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856"} err="failed to get container status \"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856\": rpc error: code = NotFound desc = could not find container \"b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856\": container with ID starting with b8b3c97ba304d823569b44621eb26e08dda7d3f9b518474e7bb8812d7dbfd856 not found: ID does not exist"
Sep 09 20:33:49.032300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.032273    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-255pp"]
Sep 09 20:33:49.125091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:49.125061    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" path="/var/lib/kubelet/pods/89c01275-441f-4afb-a7d2-6c6fd7c0b50b/volumes"
Sep 09 20:33:52.020582 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:52.020550    2582 generic.go:356] "Generic (PLEG): container finished" podID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerID="332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622" exitCode=0
Sep 09 20:33:52.021034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:52.020622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerDied","Data":"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"}
Sep 09 20:33:53.033283 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.033234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerStarted","Data":"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"}
Sep 09 20:33:53.033698 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.033388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerStarted","Data":"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"}
Sep 09 20:33:53.033771 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.033691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:53.033836 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.033771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:53.053969 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.053925    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" podStartSLOduration=3.190157827 podStartE2EDuration="7.053910802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.863752975s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:53.0519419 +0000 UTC m=+2322.452619091" watchObservedRunningTime="2026-09-09 20:33:53.053910802 +0000 UTC m=+2322.454588021"
Sep 09 20:33:53.786526 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.786490    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:33:53.786761 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.786738    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" containerID="cri-o://61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710" gracePeriod=30
Sep 09 20:33:53.786849 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:53.786777    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy" containerID="cri-o://b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566" gracePeriod=30
Sep 09 20:33:54.039471 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:54.039392    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerID="b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566" exitCode=2
Sep 09 20:33:54.039818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:54.039465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerDied","Data":"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"}
Sep 09 20:33:55.634528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:55.634487    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 09 20:33:55.639876 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:55.639847    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 09 20:33:59.045222 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.045197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:33:59.279965 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.279926    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:33:59.280328 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280314    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="storage-initializer"
Sep 09 20:33:59.280328 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="storage-initializer"
Sep 09 20:33:59.280420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280340    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kserve-container"
Sep 09 20:33:59.280420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280346    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kserve-container"
Sep 09 20:33:59.280420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280360    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kube-rbac-proxy"
Sep 09 20:33:59.280420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kube-rbac-proxy"
Sep 09 20:33:59.280556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280431    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kube-rbac-proxy"
Sep 09 20:33:59.280556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.280440    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c01275-441f-4afb-a7d2-6c6fd7c0b50b" containerName="kserve-container"
Sep 09 20:33:59.283774 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.283757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.288282 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.288259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:33:59.288396 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.288298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:33:59.303925 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.303860    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:33:59.426381 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.426349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.426554 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.426390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.426554 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.426419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fzt\" (UniqueName: \"kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.426554 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.426505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.426554 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.426547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.527922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.527888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528102 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.527941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fzt\" (UniqueName: \"kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528102 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.527985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528102 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.528008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528102 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.528082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528370 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.528349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528755 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.528736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.528801 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.528739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.530457 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.530438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.539831 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.539806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fzt\" (UniqueName: \"kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt\") pod \"isvc-canary-stop-predictor-5d5b899946-xdjl8\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.593968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.593901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:33:59.888319 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:33:59.888227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:33:59.891234 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:33:59.891207    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode564ad88_7598_4945_8875_1051019f9107.slice/crio-8faa98b4d9b39b5dc00fd485cb8d4fdc92af51ea388cfb25313e8ad4ddad1655 WatchSource:0}: Error finding container 8faa98b4d9b39b5dc00fd485cb8d4fdc92af51ea388cfb25313e8ad4ddad1655: Status 404 returned error can't find the container with id 8faa98b4d9b39b5dc00fd485cb8d4fdc92af51ea388cfb25313e8ad4ddad1655
Sep 09 20:34:00.069484 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:00.069441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerStarted","Data":"8a39e0094d1b038a6b24f6e76087ab4b254364bb96891886d43e735a08b803db"}
Sep 09 20:34:00.069894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:00.069528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerStarted","Data":"8faa98b4d9b39b5dc00fd485cb8d4fdc92af51ea388cfb25313e8ad4ddad1655"}
Sep 09 20:34:00.634626 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:00.634587    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 09 20:34:01.935522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:01.935497    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:34:02.051084 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert\") pod \"ca34996c-0b01-413a-bfbf-c5629211f4ef\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") "
Sep 09 20:34:02.051084 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051048    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config\") pod \"ca34996c-0b01-413a-bfbf-c5629211f4ef\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") "
Sep 09 20:34:02.051269 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls\") pod \"ca34996c-0b01-413a-bfbf-c5629211f4ef\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") "
Sep 09 20:34:02.051269 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051142    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrl7l\" (UniqueName: \"kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l\") pod \"ca34996c-0b01-413a-bfbf-c5629211f4ef\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") "
Sep 09 20:34:02.051269 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location\") pod \"ca34996c-0b01-413a-bfbf-c5629211f4ef\" (UID: \"ca34996c-0b01-413a-bfbf-c5629211f4ef\") "
Sep 09 20:34:02.051437 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca34996c-0b01-413a-bfbf-c5629211f4ef" (UID: "ca34996c-0b01-413a-bfbf-c5629211f4ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:02.051612 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca34996c-0b01-413a-bfbf-c5629211f4ef" (UID: "ca34996c-0b01-413a-bfbf-c5629211f4ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:02.051612 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.051593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca34996c-0b01-413a-bfbf-c5629211f4ef" (UID: "ca34996c-0b01-413a-bfbf-c5629211f4ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:02.053210 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.053188    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l" (OuterVolumeSpecName: "kube-api-access-mrl7l") pod "ca34996c-0b01-413a-bfbf-c5629211f4ef" (UID: "ca34996c-0b01-413a-bfbf-c5629211f4ef"). InnerVolumeSpecName "kube-api-access-mrl7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:02.053267 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.053211    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca34996c-0b01-413a-bfbf-c5629211f4ef" (UID: "ca34996c-0b01-413a-bfbf-c5629211f4ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:02.082999 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.082972    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerID="61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710" exitCode=0
Sep 09 20:34:02.083106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.083023    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"
Sep 09 20:34:02.083106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.083044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerDied","Data":"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"}
Sep 09 20:34:02.083106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.083096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8" event={"ID":"ca34996c-0b01-413a-bfbf-c5629211f4ef","Type":"ContainerDied","Data":"da300988c6440b4eaee9e371d99acc4c08564489f1e4b2b36f7cff963b235863"}
Sep 09 20:34:02.083222 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.083123    2582 scope.go:117] "RemoveContainer" containerID="b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"
Sep 09 20:34:02.091894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.091878    2582 scope.go:117] "RemoveContainer" containerID="61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"
Sep 09 20:34:02.099119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.099103    2582 scope.go:117] "RemoveContainer" containerID="6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"
Sep 09 20:34:02.107615 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.107594    2582 scope.go:117] "RemoveContainer" containerID="b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"
Sep 09 20:34:02.107945 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.107905    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:34:02.108091 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:02.107966    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566\": container with ID starting with b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566 not found: ID does not exist" containerID="b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"
Sep 09 20:34:02.108091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.107998    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566"} err="failed to get container status \"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566\": rpc error: code = NotFound desc = could not find container \"b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566\": container with ID starting with b3e32b5c3b5beac1f4651f387074d8c1d47795af2b131d691ed6263fe108c566 not found: ID does not exist"
Sep 09 20:34:02.108091 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.108028    2582 scope.go:117] "RemoveContainer" containerID="61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"
Sep 09 20:34:02.108620 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:02.108599    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710\": container with ID starting with 61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710 not found: ID does not exist" containerID="61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"
Sep 09 20:34:02.108896 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.108624    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710"} err="failed to get container status \"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710\": rpc error: code = NotFound desc = could not find container \"61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710\": container with ID starting with 61cc9ec1a173097f9b4f4a8bd526e6df936c86e40194486f352aaf0172b4c710 not found: ID does not exist"
Sep 09 20:34:02.108896 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.108644    2582 scope.go:117] "RemoveContainer" containerID="6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"
Sep 09 20:34:02.109010 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:02.108924    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540\": container with ID starting with 6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540 not found: ID does not exist" containerID="6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"
Sep 09 20:34:02.109010 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.108953    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540"} err="failed to get container status \"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540\": rpc error: code = NotFound desc = could not find container \"6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540\": container with ID starting with 6e0e0d5ec11d2fbbdba03ac1ea2e9e24c8926eaf3083a7156b0f98c2e4326540 not found: ID does not exist"
Sep 09 20:34:02.110516 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.110492    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-v4ns8"]
Sep 09 20:34:02.152786 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.152763    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:02.152786 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.152784    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:02.152917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.152798    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca34996c-0b01-413a-bfbf-c5629211f4ef-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:02.152917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.152812    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrl7l\" (UniqueName: \"kubernetes.io/projected/ca34996c-0b01-413a-bfbf-c5629211f4ef-kube-api-access-mrl7l\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:02.152917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:02.152824    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca34996c-0b01-413a-bfbf-c5629211f4ef-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:03.123917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:03.123882    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" path="/var/lib/kubelet/pods/ca34996c-0b01-413a-bfbf-c5629211f4ef/volumes"
Sep 09 20:34:04.095538 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:04.095454    2582 generic.go:356] "Generic (PLEG): container finished" podID="e564ad88-7598-4945-8875-1051019f9107" containerID="8a39e0094d1b038a6b24f6e76087ab4b254364bb96891886d43e735a08b803db" exitCode=0
Sep 09 20:34:04.095538 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:04.095524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerDied","Data":"8a39e0094d1b038a6b24f6e76087ab4b254364bb96891886d43e735a08b803db"}
Sep 09 20:34:05.102728 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:05.102693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerStarted","Data":"79b946a728cc3ea9a48d5b0218a3e14f7dce9539be5be2fe25f8363e37230c1e"}
Sep 09 20:34:05.103190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:05.102763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerStarted","Data":"1b3d3f1fade43351c5adbe71d87557effd386666f2052d389227189cbfa7cae1"}
Sep 09 20:34:05.103190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:05.102976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:34:05.125578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:05.125527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podStartSLOduration=2.273076268 podStartE2EDuration="6.125512827s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.852436559s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:05.124263239 +0000 UTC m=+2334.524940431" watchObservedRunningTime="2026-09-09 20:34:05.125512827 +0000 UTC m=+2334.526190018"
Sep 09 20:34:06.110359 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:06.110321    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:34:06.111627 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:06.111599    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:07.115318 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:07.115277    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:12.120559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:12.120527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:34:12.121003 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:12.120979    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:22.121371 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:22.121330    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:29.074767 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:29.074669    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:34:32.121897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:32.121861    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:36.976183 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:36.976152    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:34:36.976697 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:36.976398    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kserve-container" containerID="cri-o://77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932" gracePeriod=30
Sep 09 20:34:36.976697 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:36.976463    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kube-rbac-proxy" containerID="cri-o://1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0" gracePeriod=30
Sep 09 20:34:37.253464 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:37.253376    2582 generic.go:356] "Generic (PLEG): container finished" podID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerID="1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0" exitCode=2
Sep 09 20:34:37.253464 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:37.253418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerDied","Data":"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"}
Sep 09 20:34:39.040754 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:39.040715    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 09 20:34:42.120974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.120936    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:34:42.149558 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149500    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:34:42.149902 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149889    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container"
Sep 09 20:34:42.149960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149903    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container"
Sep 09 20:34:42.149960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149914    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="storage-initializer"
Sep 09 20:34:42.149960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149920    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="storage-initializer"
Sep 09 20:34:42.149960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149932    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy"
Sep 09 20:34:42.149960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149938    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy"
Sep 09 20:34:42.150119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.149996    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kserve-container"
Sep 09 20:34:42.150119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.150005    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca34996c-0b01-413a-bfbf-c5629211f4ef" containerName="kube-rbac-proxy"
Sep 09 20:34:42.153242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.153215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.155484 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.155466    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:34:42.155579 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.155540    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 09 20:34:42.163764 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.163740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:34:42.211260 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.211219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlbx\" (UniqueName: \"kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.211429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.211280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.211429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.211370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.211429 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.211405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.211582 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.211567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.312659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.312628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.312659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.312661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.312907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.312807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.312907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.312857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlbx\" (UniqueName: \"kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.313011 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.312914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.313150 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.313128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.313397 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.313380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.313510 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.313492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.315287 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.315264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.320928 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.320903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlbx\" (UniqueName: \"kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-g8pgf\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.463864 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.463839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:34:42.744664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:42.737981    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:34:43.287447 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.287405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerStarted","Data":"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"}
Sep 09 20:34:43.287850 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.287506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerStarted","Data":"a0bebd39e270e68f50210cfa6ac7b8ab02db77d2b9ea7c8bd3553d2012fe702a"}
Sep 09 20:34:43.445448 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.445420    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:34:43.523914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.523882    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert\") pod \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") "
Sep 09 20:34:43.524042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.523939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlc6g\" (UniqueName: \"kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g\") pod \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") "
Sep 09 20:34:43.524042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.523970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config\") pod \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") "
Sep 09 20:34:43.524042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.524004    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location\") pod \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") "
Sep 09 20:34:43.524042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.524031    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls\") pod \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\" (UID: \"7960bebc-b48c-4f12-a2dc-48acf55fbe47\") "
Sep 09 20:34:43.524323 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.524289    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7960bebc-b48c-4f12-a2dc-48acf55fbe47" (UID: "7960bebc-b48c-4f12-a2dc-48acf55fbe47"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:43.524440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.524317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7960bebc-b48c-4f12-a2dc-48acf55fbe47" (UID: "7960bebc-b48c-4f12-a2dc-48acf55fbe47"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:43.524440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.524350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7960bebc-b48c-4f12-a2dc-48acf55fbe47" (UID: "7960bebc-b48c-4f12-a2dc-48acf55fbe47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:43.526093 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.526071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g" (OuterVolumeSpecName: "kube-api-access-dlc6g") pod "7960bebc-b48c-4f12-a2dc-48acf55fbe47" (UID: "7960bebc-b48c-4f12-a2dc-48acf55fbe47"). InnerVolumeSpecName "kube-api-access-dlc6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:43.526180 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.526112    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7960bebc-b48c-4f12-a2dc-48acf55fbe47" (UID: "7960bebc-b48c-4f12-a2dc-48acf55fbe47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:43.625618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.625582    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlc6g\" (UniqueName: \"kubernetes.io/projected/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-api-access-dlc6g\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:43.625618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.625612    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:43.625813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.625626    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7960bebc-b48c-4f12-a2dc-48acf55fbe47-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:43.625813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.625640    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7960bebc-b48c-4f12-a2dc-48acf55fbe47-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:43.625813 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:43.625654    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7960bebc-b48c-4f12-a2dc-48acf55fbe47-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:34:44.294888 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.294855    2582 generic.go:356] "Generic (PLEG): container finished" podID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerID="77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932" exitCode=0
Sep 09 20:34:44.295303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.294916    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"
Sep 09 20:34:44.295303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.294939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerDied","Data":"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"}
Sep 09 20:34:44.295303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.295002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk" event={"ID":"7960bebc-b48c-4f12-a2dc-48acf55fbe47","Type":"ContainerDied","Data":"aef2554d1d9d7a5edb48d28f232495a51583143ae08603b6e552f2cfb8263be4"}
Sep 09 20:34:44.295303 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.295031    2582 scope.go:117] "RemoveContainer" containerID="1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"
Sep 09 20:34:44.304113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.304095    2582 scope.go:117] "RemoveContainer" containerID="77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"
Sep 09 20:34:44.311794 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.311779    2582 scope.go:117] "RemoveContainer" containerID="332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"
Sep 09 20:34:44.317524 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.317501    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:34:44.320017 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.320000    2582 scope.go:117] "RemoveContainer" containerID="1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"
Sep 09 20:34:44.320352 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:44.320330    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0\": container with ID starting with 1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0 not found: ID does not exist" containerID="1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"
Sep 09 20:34:44.320432 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.320367    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0"} err="failed to get container status \"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0\": rpc error: code = NotFound desc = could not find container \"1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0\": container with ID starting with 1b3a66b44f571bb0c536ad2a3943abd0dcb1cb33fe93bdde59ac57a285f544b0 not found: ID does not exist"
Sep 09 20:34:44.320432 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.320392    2582 scope.go:117] "RemoveContainer" containerID="77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"
Sep 09 20:34:44.320711 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:44.320672    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932\": container with ID starting with 77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932 not found: ID does not exist" containerID="77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"
Sep 09 20:34:44.320763 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.320720    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932"} err="failed to get container status \"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932\": rpc error: code = NotFound desc = could not find container \"77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932\": container with ID starting with 77250f6bef8498e4f4e36d03ef4ffc1a44804685d27c4a31a0fe65335efe8932 not found: ID does not exist"
Sep 09 20:34:44.320763 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.320735    2582 scope.go:117] "RemoveContainer" containerID="332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"
Sep 09 20:34:44.321001 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:34:44.320984    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622\": container with ID starting with 332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622 not found: ID does not exist" containerID="332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"
Sep 09 20:34:44.321045 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.321010    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622"} err="failed to get container status \"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622\": rpc error: code = NotFound desc = could not find container \"332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622\": container with ID starting with 332a30d03a7d8b6e20e3237ebde2ef456b762bc601860a5c378b38b9751fc622 not found: ID does not exist"
Sep 09 20:34:44.321240 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:44.321225    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-jxslk"]
Sep 09 20:34:45.125486 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:45.125443    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" path="/var/lib/kubelet/pods/7960bebc-b48c-4f12-a2dc-48acf55fbe47/volumes"
Sep 09 20:34:47.311414 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:47.311381    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerID="cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8" exitCode=0
Sep 09 20:34:47.311414 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:47.311418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerDied","Data":"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"}
Sep 09 20:34:47.312613 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:47.312595    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:34:52.121219 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:34:52.121174    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:35:02.121128 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:02.121080    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:35:06.417355 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:06.417321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerStarted","Data":"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"}
Sep 09 20:35:06.417846 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:06.417393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerStarted","Data":"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"}
Sep 09 20:35:06.417846 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:06.417622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:35:06.443944 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:06.443899    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podStartSLOduration=2.361175183 podStartE2EDuration="24.443883606s" totalImagesPullingTime="18.496252341s" totalInitContainerRuntime="3.586456082s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:06.441649788 +0000 UTC m=+2395.842326983" watchObservedRunningTime="2026-09-09 20:35:06.443883606 +0000 UTC m=+2395.844560797"
Sep 09 20:35:07.421941 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:07.421910    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:35:07.423388 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:07.423364    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:08.426618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:08.426573    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:12.121195 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:12.121157    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 09 20:35:13.431916 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:13.431886    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:35:13.432375 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:13.432347    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:21.007658 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.007620    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:35:21.008080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.007874    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container" containerID="cri-o://1b3d3f1fade43351c5adbe71d87557effd386666f2052d389227189cbfa7cae1" gracePeriod=30
Sep 09 20:35:21.008080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.007971    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kube-rbac-proxy" containerID="cri-o://79b946a728cc3ea9a48d5b0218a3e14f7dce9539be5be2fe25f8363e37230c1e" gracePeriod=30
Sep 09 20:35:21.008875 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.008844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:35:21.487253 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.487220    2582 generic.go:356] "Generic (PLEG): container finished" podID="e564ad88-7598-4945-8875-1051019f9107" containerID="79b946a728cc3ea9a48d5b0218a3e14f7dce9539be5be2fe25f8363e37230c1e" exitCode=2
Sep 09 20:35:21.487434 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:21.487302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerDied","Data":"79b946a728cc3ea9a48d5b0218a3e14f7dce9539be5be2fe25f8363e37230c1e"}
Sep 09 20:35:22.116595 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:22.116555    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 09 20:35:23.433115 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:23.433079    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:25.507567 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.507539    2582 generic.go:356] "Generic (PLEG): container finished" podID="e564ad88-7598-4945-8875-1051019f9107" containerID="1b3d3f1fade43351c5adbe71d87557effd386666f2052d389227189cbfa7cae1" exitCode=0
Sep 09 20:35:25.507954 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.507606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerDied","Data":"1b3d3f1fade43351c5adbe71d87557effd386666f2052d389227189cbfa7cae1"}
Sep 09 20:35:25.554284 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.554259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:35:25.610593 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config\") pod \"e564ad88-7598-4945-8875-1051019f9107\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") "
Sep 09 20:35:25.610766 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert\") pod \"e564ad88-7598-4945-8875-1051019f9107\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") "
Sep 09 20:35:25.610766 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls\") pod \"e564ad88-7598-4945-8875-1051019f9107\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") "
Sep 09 20:35:25.610766 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location\") pod \"e564ad88-7598-4945-8875-1051019f9107\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") "
Sep 09 20:35:25.610944 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610768    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2fzt\" (UniqueName: \"kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt\") pod \"e564ad88-7598-4945-8875-1051019f9107\" (UID: \"e564ad88-7598-4945-8875-1051019f9107\") "
Sep 09 20:35:25.611024 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.610999    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e564ad88-7598-4945-8875-1051019f9107" (UID: "e564ad88-7598-4945-8875-1051019f9107"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:25.611109 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.611088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e564ad88-7598-4945-8875-1051019f9107" (UID: "e564ad88-7598-4945-8875-1051019f9107"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:25.611175 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.611103    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e564ad88-7598-4945-8875-1051019f9107" (UID: "e564ad88-7598-4945-8875-1051019f9107"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:25.612733 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.612703    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e564ad88-7598-4945-8875-1051019f9107" (UID: "e564ad88-7598-4945-8875-1051019f9107"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:25.612825 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.612747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt" (OuterVolumeSpecName: "kube-api-access-l2fzt") pod "e564ad88-7598-4945-8875-1051019f9107" (UID: "e564ad88-7598-4945-8875-1051019f9107"). InnerVolumeSpecName "kube-api-access-l2fzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:25.711913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.711886    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:35:25.711913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.711910    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e564ad88-7598-4945-8875-1051019f9107-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:35:25.712083 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.711919    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e564ad88-7598-4945-8875-1051019f9107-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:35:25.712083 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.711930    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fzt\" (UniqueName: \"kubernetes.io/projected/e564ad88-7598-4945-8875-1051019f9107-kube-api-access-l2fzt\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:35:25.712083 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:25.711938    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e564ad88-7598-4945-8875-1051019f9107-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:35:26.513664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.513626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8" event={"ID":"e564ad88-7598-4945-8875-1051019f9107","Type":"ContainerDied","Data":"8faa98b4d9b39b5dc00fd485cb8d4fdc92af51ea388cfb25313e8ad4ddad1655"}
Sep 09 20:35:26.514082 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.513668    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"
Sep 09 20:35:26.514082 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.513730    2582 scope.go:117] "RemoveContainer" containerID="79b946a728cc3ea9a48d5b0218a3e14f7dce9539be5be2fe25f8363e37230c1e"
Sep 09 20:35:26.522365 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.522342    2582 scope.go:117] "RemoveContainer" containerID="1b3d3f1fade43351c5adbe71d87557effd386666f2052d389227189cbfa7cae1"
Sep 09 20:35:26.529541 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.529523    2582 scope.go:117] "RemoveContainer" containerID="8a39e0094d1b038a6b24f6e76087ab4b254364bb96891886d43e735a08b803db"
Sep 09 20:35:26.536602 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.536581    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:35:26.541260 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:26.541235    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-xdjl8"]
Sep 09 20:35:27.124331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:27.124295    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e564ad88-7598-4945-8875-1051019f9107" path="/var/lib/kubelet/pods/e564ad88-7598-4945-8875-1051019f9107/volumes"
Sep 09 20:35:33.433067 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:33.433031    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:39.343014 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.342974    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343416    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="storage-initializer"
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343431    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="storage-initializer"
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343444    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="storage-initializer"
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343449    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="storage-initializer"
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343465    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kube-rbac-proxy"
Sep 09 20:35:39.343473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343471    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kube-rbac-proxy"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343484    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343489    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343497    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343503    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343509    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kube-rbac-proxy"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343514    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kube-rbac-proxy"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343563    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343571    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kube-rbac-proxy"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343577    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7960bebc-b48c-4f12-a2dc-48acf55fbe47" containerName="kserve-container"
Sep 09 20:35:39.343668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.343584    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564ad88-7598-4945-8875-1051019f9107" containerName="kube-rbac-proxy"
Sep 09 20:35:39.495726 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.495673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:35:39.495890 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.495796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.499155 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.499124    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 09 20:35:39.499274 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.499254    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 20:35:39.636098 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.636012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.636098 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.636057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.636331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.636117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.636331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.636187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.636331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.636242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kjj\" (UniqueName: \"kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.736874 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.736832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.736892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kjj\" (UniqueName: \"kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.736921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.736950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.736977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737261 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:35:39.737086    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 09 20:35:39.737261 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:35:39.737147    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls podName:bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83 nodeName:}" failed. No retries permitted until 2026-09-09 20:35:40.237126695 +0000 UTC m=+2429.637803865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-4vwzg" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 09 20:35:39.737373 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.737278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737488 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.737470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.737568 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.737553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:39.748675 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:39.748648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kjj\" (UniqueName: \"kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:40.241798 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:40.241759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:40.244106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:40.244075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-4vwzg\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:40.406172 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:40.406136    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:40.688629 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:40.688604    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:35:40.691279 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:35:40.691242    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc9996a_b4bc_4c6b_80e1_d6ff34b6fc83.slice/crio-bf87746f0657afa57da24252eceaf6c7c7b77ea052c63da04986130d07d567f5 WatchSource:0}: Error finding container bf87746f0657afa57da24252eceaf6c7c7b77ea052c63da04986130d07d567f5: Status 404 returned error can't find the container with id bf87746f0657afa57da24252eceaf6c7c7b77ea052c63da04986130d07d567f5
Sep 09 20:35:41.588600 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:41.588562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerStarted","Data":"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"}
Sep 09 20:35:41.588996 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:41.588647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerStarted","Data":"bf87746f0657afa57da24252eceaf6c7c7b77ea052c63da04986130d07d567f5"}
Sep 09 20:35:43.433349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:43.433307    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:45.608064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:45.608029    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerID="834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d" exitCode=0
Sep 09 20:35:45.608064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:45.608066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerDied","Data":"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"}
Sep 09 20:35:53.432640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:53.432602    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:35:53.651349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:53.651316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerStarted","Data":"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"}
Sep 09 20:35:53.651518 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:53.651380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerStarted","Data":"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"}
Sep 09 20:35:53.651596 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:53.651575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:53.680203 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:53.680139    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podStartSLOduration=3.5947918210000003 podStartE2EDuration="14.680124379s" totalImagesPullingTime="7.139318747s" totalInitContainerRuntime="3.946013811s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:53.676399505 +0000 UTC m=+2443.077076697" watchObservedRunningTime="2026-09-09 20:35:53.680124379 +0000 UTC m=+2443.080801571"
Sep 09 20:35:54.656069 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:54.656038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:35:54.656934 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:54.656911    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:35:55.661064 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:35:55.661027    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:00.665886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:00.665849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:36:00.666368 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:00.666331    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:03.432485 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:03.432442    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:36:10.666278 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:10.666236    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:13.433741 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:13.433703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:36:20.666855 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:20.666811    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:22.204068 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:22.204028    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:36:22.204742 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:22.204691    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" containerID="cri-o://7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3" gracePeriod=30
Sep 09 20:36:22.204906 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:22.204831    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kube-rbac-proxy" containerID="cri-o://a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005" gracePeriod=30
Sep 09 20:36:22.794259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:22.794226    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerID="a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005" exitCode=2
Sep 09 20:36:22.794430 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:22.794295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerDied","Data":"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"}
Sep 09 20:36:23.427358 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:23.427313    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 09 20:36:23.432671 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:23.432637    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 09 20:36:26.163999 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.163977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:36:26.235640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235562    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls\") pod \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") "
Sep 09 20:36:26.235640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config\") pod \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") "
Sep 09 20:36:26.235640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location\") pod \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") "
Sep 09 20:36:26.235929 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert\") pod \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") "
Sep 09 20:36:26.235929 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vlbx\" (UniqueName: \"kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx\") pod \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\" (UID: \"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65\") "
Sep 09 20:36:26.236044 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.235992    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" (UID: "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:26.236044 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.236020    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" (UID: "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:26.236116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.236037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" (UID: "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:26.237734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.237710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx" (OuterVolumeSpecName: "kube-api-access-5vlbx") pod "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" (UID: "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65"). InnerVolumeSpecName "kube-api-access-5vlbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:26.237821 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.237788    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" (UID: "a0a23ff9-2ff4-4e04-9f16-bc161b3ded65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:26.336269 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.336244    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:36:26.336269 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.336268    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlbx\" (UniqueName: \"kubernetes.io/projected/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-api-access-5vlbx\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:36:26.336411 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.336278    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:36:26.336411 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.336286    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:36:26.336411 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.336295    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:36:26.813209 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.813177    2582 generic.go:356] "Generic (PLEG): container finished" podID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerID="7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3" exitCode=0
Sep 09 20:36:26.813403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.813235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerDied","Data":"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"}
Sep 09 20:36:26.813403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.813240    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"
Sep 09 20:36:26.813403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.813281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf" event={"ID":"a0a23ff9-2ff4-4e04-9f16-bc161b3ded65","Type":"ContainerDied","Data":"a0bebd39e270e68f50210cfa6ac7b8ab02db77d2b9ea7c8bd3553d2012fe702a"}
Sep 09 20:36:26.813403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.813310    2582 scope.go:117] "RemoveContainer" containerID="a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"
Sep 09 20:36:26.821842 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.821821    2582 scope.go:117] "RemoveContainer" containerID="7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"
Sep 09 20:36:26.829707 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.829690    2582 scope.go:117] "RemoveContainer" containerID="cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"
Sep 09 20:36:26.836512 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.836488    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:36:26.837329 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.837314    2582 scope.go:117] "RemoveContainer" containerID="a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"
Sep 09 20:36:26.837617 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:36:26.837599    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005\": container with ID starting with a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005 not found: ID does not exist" containerID="a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"
Sep 09 20:36:26.837700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.837625    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005"} err="failed to get container status \"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005\": rpc error: code = NotFound desc = could not find container \"a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005\": container with ID starting with a9041c1cccd765041ece1f1de224c3492b5351404853854dc62e13f29b44a005 not found: ID does not exist"
Sep 09 20:36:26.837700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.837640    2582 scope.go:117] "RemoveContainer" containerID="7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"
Sep 09 20:36:26.838070 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:36:26.838043    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3\": container with ID starting with 7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3 not found: ID does not exist" containerID="7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"
Sep 09 20:36:26.838197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.838078    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3"} err="failed to get container status \"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3\": rpc error: code = NotFound desc = could not find container \"7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3\": container with ID starting with 7c6d2f0604d8be41f68a2e88a974d8b5bb437438000cee80de5c781b0a4967c3 not found: ID does not exist"
Sep 09 20:36:26.838197 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.838101    2582 scope.go:117] "RemoveContainer" containerID="cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"
Sep 09 20:36:26.838706 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:36:26.838661    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8\": container with ID starting with cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8 not found: ID does not exist" containerID="cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"
Sep 09 20:36:26.839088 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.839064    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8"} err="failed to get container status \"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8\": rpc error: code = NotFound desc = could not find container \"cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8\": container with ID starting with cf2a343d2ab2bccb2aaeb526069021c8e7bae1d04861706cc995fccac04244b8 not found: ID does not exist"
Sep 09 20:36:26.840290 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:26.840273    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-g8pgf"]
Sep 09 20:36:27.124443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.124361    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" path="/var/lib/kubelet/pods/a0a23ff9-2ff4-4e04-9f16-bc161b3ded65/volumes"
Sep 09 20:36:27.473421 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473387    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473785    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="storage-initializer"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473799    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="storage-initializer"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473810    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473816    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473831    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kube-rbac-proxy"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kube-rbac-proxy"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473897    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kserve-container"
Sep 09 20:36:27.473922 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.473908    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a23ff9-2ff4-4e04-9f16-bc161b3ded65" containerName="kube-rbac-proxy"
Sep 09 20:36:27.477667 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.477269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.480938 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.480641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 09 20:36:27.480938 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.480676    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:36:27.488543 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.488522    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:36:27.545313 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.545275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.545482 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.545327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.545482 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.545407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdcj\" (UniqueName: \"kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.545482 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.545450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.545640 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.545497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.646673 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.646638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdcj\" (UniqueName: \"kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.646884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.646705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.646884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.646749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.646884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.646826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.647048 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.646960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.647191 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.647171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.647528 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.647504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.647648 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.647532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.649288 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.649269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.655165 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.655135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdcj\" (UniqueName: \"kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:27.789746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:27.789630    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:28.061904 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:28.061876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:36:28.064131 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:36:28.064104    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dec7b4_03fc_4638_8b47_42f02fb537a8.slice/crio-c1382445423457831f603aa21cf0f6faa3f82705234f855a193c453dabee3355 WatchSource:0}: Error finding container c1382445423457831f603aa21cf0f6faa3f82705234f855a193c453dabee3355: Status 404 returned error can't find the container with id c1382445423457831f603aa21cf0f6faa3f82705234f855a193c453dabee3355
Sep 09 20:36:28.825063 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:28.825026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerStarted","Data":"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"}
Sep 09 20:36:28.825426 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:28.825118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerStarted","Data":"c1382445423457831f603aa21cf0f6faa3f82705234f855a193c453dabee3355"}
Sep 09 20:36:30.666776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:30.666733    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:32.844259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:32.844217    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerID="5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca" exitCode=0
Sep 09 20:36:32.844633 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:32.844293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerDied","Data":"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"}
Sep 09 20:36:33.851284 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:33.851245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerStarted","Data":"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"}
Sep 09 20:36:33.851664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:33.851315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerStarted","Data":"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"}
Sep 09 20:36:33.851664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:33.851503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:33.871388 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:33.871343    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podStartSLOduration=3.046679709 podStartE2EDuration="6.871328837s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824649128s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:33.870022177 +0000 UTC m=+2483.270699368" watchObservedRunningTime="2026-09-09 20:36:33.871328837 +0000 UTC m=+2483.272006029"
Sep 09 20:36:34.855947 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:34.855910    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:40.666268 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:40.666228    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:36:40.865387 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:40.865358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:36:50.666900 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:36:50.666861    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:37:00.667170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:00.667123    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:37:10.667713 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:10.667615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:37:10.873241 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:10.873198    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:37:19.364114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:19.364081    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:37:19.364504 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:19.364323    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" containerID="cri-o://d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96" gracePeriod=30
Sep 09 20:37:19.364504 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:19.364372    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kube-rbac-proxy" containerID="cri-o://5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133" gracePeriod=30
Sep 09 20:37:20.067573 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:20.067535    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerID="5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133" exitCode=2
Sep 09 20:37:20.067755 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:20.067604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerDied","Data":"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"}
Sep 09 20:37:20.661815 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:20.661767    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 09 20:37:20.667192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:20.667159    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 09 20:37:20.867821 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:20.867788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:37:23.903170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.903137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:37:23.951220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") pod \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") "
Sep 09 20:37:23.951220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951197    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7kjj\" (UniqueName: \"kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj\") pod \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") "
Sep 09 20:37:23.951417 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location\") pod \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") "
Sep 09 20:37:23.951417 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951251    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config\") pod \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") "
Sep 09 20:37:23.951417 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951302    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert\") pod \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\" (UID: \"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83\") "
Sep 09 20:37:23.951630 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:23.951729 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:23.951780 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.951726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:23.953246 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.953225    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj" (OuterVolumeSpecName: "kube-api-access-m7kjj") pod "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83"). InnerVolumeSpecName "kube-api-access-m7kjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:23.953308 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:23.953259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" (UID: "bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:24.052290 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.052245    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:24.052290 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.052287    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:24.052473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.052301    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kjj\" (UniqueName: \"kubernetes.io/projected/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-api-access-m7kjj\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:24.052473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.052313    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:24.052473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.052326    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:24.089493 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.089459    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerID="d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96" exitCode=0
Sep 09 20:37:24.089629 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.089520    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"
Sep 09 20:37:24.089629 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.089544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerDied","Data":"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"}
Sep 09 20:37:24.089629 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.089607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg" event={"ID":"bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83","Type":"ContainerDied","Data":"bf87746f0657afa57da24252eceaf6c7c7b77ea052c63da04986130d07d567f5"}
Sep 09 20:37:24.089772 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.089633    2582 scope.go:117] "RemoveContainer" containerID="5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"
Sep 09 20:37:24.098338 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.098318    2582 scope.go:117] "RemoveContainer" containerID="d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"
Sep 09 20:37:24.105637 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.105623    2582 scope.go:117] "RemoveContainer" containerID="834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"
Sep 09 20:37:24.111060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.111035    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:37:24.113644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.113623    2582 scope.go:117] "RemoveContainer" containerID="5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"
Sep 09 20:37:24.113946 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:24.113929    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133\": container with ID starting with 5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133 not found: ID does not exist" containerID="5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"
Sep 09 20:37:24.114022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.113957    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133"} err="failed to get container status \"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133\": rpc error: code = NotFound desc = could not find container \"5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133\": container with ID starting with 5bc03fd5e9271d1c67640f94fb26634f3b5194b7ee2a6ff30da6b4beed6bf133 not found: ID does not exist"
Sep 09 20:37:24.114022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.113974    2582 scope.go:117] "RemoveContainer" containerID="d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"
Sep 09 20:37:24.114248 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:24.114230    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96\": container with ID starting with d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96 not found: ID does not exist" containerID="d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"
Sep 09 20:37:24.114322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.114256    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96"} err="failed to get container status \"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96\": rpc error: code = NotFound desc = could not find container \"d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96\": container with ID starting with d274d98433b1f3f66e353dfe4025ecd4b74969155715a4f5bd9a952437b70f96 not found: ID does not exist"
Sep 09 20:37:24.114322 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.114276    2582 scope.go:117] "RemoveContainer" containerID="834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"
Sep 09 20:37:24.114542 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:24.114523    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d\": container with ID starting with 834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d not found: ID does not exist" containerID="834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"
Sep 09 20:37:24.114582 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.114547    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d"} err="failed to get container status \"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d\": rpc error: code = NotFound desc = could not find container \"834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d\": container with ID starting with 834856e9d463ae556564ec6dcdf038f60fcd1dd8b200268bf13d8758e981378d not found: ID does not exist"
Sep 09 20:37:24.115079 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:24.115063    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-4vwzg"]
Sep 09 20:37:25.123900 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:25.123867    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" path="/var/lib/kubelet/pods/bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83/volumes"
Sep 09 20:37:27.493338 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:27.493308    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:37:27.493815 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:27.493551    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container" containerID="cri-o://9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd" gracePeriod=30
Sep 09 20:37:27.493815 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:27.493593    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kube-rbac-proxy" containerID="cri-o://40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6" gracePeriod=30
Sep 09 20:37:28.111841 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:28.111807    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerID="40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6" exitCode=2
Sep 09 20:37:28.112057 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:28.111881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerDied","Data":"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"}
Sep 09 20:37:30.860473 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:30.860431    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 09 20:37:30.866129 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:30.866097    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 09 20:37:34.542646 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.542622    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:37:34.642263 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert\") pod \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") "
Sep 09 20:37:34.642263 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config\") pod \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") "
Sep 09 20:37:34.642263 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsdcj\" (UniqueName: \"kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj\") pod \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") "
Sep 09 20:37:34.642523 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls\") pod \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") "
Sep 09 20:37:34.642523 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location\") pod \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\" (UID: \"e4dec7b4-03fc-4638-8b47-42f02fb537a8\") "
Sep 09 20:37:34.642697 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4dec7b4-03fc-4638-8b47-42f02fb537a8" (UID: "e4dec7b4-03fc-4638-8b47-42f02fb537a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:34.642780 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642724    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4dec7b4-03fc-4638-8b47-42f02fb537a8" (UID: "e4dec7b4-03fc-4638-8b47-42f02fb537a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:34.642842 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.642820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4dec7b4-03fc-4638-8b47-42f02fb537a8" (UID: "e4dec7b4-03fc-4638-8b47-42f02fb537a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:34.644304 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.644283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4dec7b4-03fc-4638-8b47-42f02fb537a8" (UID: "e4dec7b4-03fc-4638-8b47-42f02fb537a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:34.644385 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.644346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj" (OuterVolumeSpecName: "kube-api-access-vsdcj") pod "e4dec7b4-03fc-4638-8b47-42f02fb537a8" (UID: "e4dec7b4-03fc-4638-8b47-42f02fb537a8"). InnerVolumeSpecName "kube-api-access-vsdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:34.743403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.743372    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:34.743403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.743400    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:34.743562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.743414    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdcj\" (UniqueName: \"kubernetes.io/projected/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kube-api-access-vsdcj\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:34.743562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.743427    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4dec7b4-03fc-4638-8b47-42f02fb537a8-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:34.743562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:34.743440    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4dec7b4-03fc-4638-8b47-42f02fb537a8-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:37:35.147482 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.147451    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerID="9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd" exitCode=0
Sep 09 20:37:35.147622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.147511    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"
Sep 09 20:37:35.147622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.147533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerDied","Data":"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"}
Sep 09 20:37:35.147622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.147591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc" event={"ID":"e4dec7b4-03fc-4638-8b47-42f02fb537a8","Type":"ContainerDied","Data":"c1382445423457831f603aa21cf0f6faa3f82705234f855a193c453dabee3355"}
Sep 09 20:37:35.147622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.147618    2582 scope.go:117] "RemoveContainer" containerID="40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"
Sep 09 20:37:35.158432 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.158411    2582 scope.go:117] "RemoveContainer" containerID="9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"
Sep 09 20:37:35.165194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.165171    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:37:35.166623 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.166609    2582 scope.go:117] "RemoveContainer" containerID="5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"
Sep 09 20:37:35.169190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.169170    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-kxzbc"]
Sep 09 20:37:35.174121 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174107    2582 scope.go:117] "RemoveContainer" containerID="40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"
Sep 09 20:37:35.174360 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:35.174341    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6\": container with ID starting with 40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6 not found: ID does not exist" containerID="40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"
Sep 09 20:37:35.174426 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174366    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6"} err="failed to get container status \"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6\": rpc error: code = NotFound desc = could not find container \"40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6\": container with ID starting with 40f3b4fdbf4bd8db894b55ce91d7eebb4919f2aa4f9bca316ec61762d3e224b6 not found: ID does not exist"
Sep 09 20:37:35.174426 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174382    2582 scope.go:117] "RemoveContainer" containerID="9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"
Sep 09 20:37:35.174618 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:35.174599    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd\": container with ID starting with 9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd not found: ID does not exist" containerID="9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"
Sep 09 20:37:35.174661 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174625    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd"} err="failed to get container status \"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd\": rpc error: code = NotFound desc = could not find container \"9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd\": container with ID starting with 9404ea206d868c2e28e5d5b205b62326a8b5d10be196960345c0186f15bf56dd not found: ID does not exist"
Sep 09 20:37:35.174661 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174640    2582 scope.go:117] "RemoveContainer" containerID="5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"
Sep 09 20:37:35.174901 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:37:35.174885    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca\": container with ID starting with 5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca not found: ID does not exist" containerID="5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"
Sep 09 20:37:35.174943 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:35.174904    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca"} err="failed to get container status \"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca\": rpc error: code = NotFound desc = could not find container \"5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca\": container with ID starting with 5e921067824dcea4c84b72f3c95923dad9c5def8b83586f70d0d1567e3c806ca not found: ID does not exist"
Sep 09 20:37:37.124177 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:37:37.124138    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" path="/var/lib/kubelet/pods/e4dec7b4-03fc-4638-8b47-42f02fb537a8/volumes"
Sep 09 20:38:47.905930 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.905896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906232    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906243    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906257    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="storage-initializer"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906262    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="storage-initializer"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906270    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kube-rbac-proxy"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kube-rbac-proxy"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906282    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906287    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906294    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kube-rbac-proxy"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906299    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kube-rbac-proxy"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906306    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="storage-initializer"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906312    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="storage-initializer"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906375    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kserve-container"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906383    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kube-rbac-proxy"
Sep 09 20:38:47.906389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906393    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dec7b4-03fc-4638-8b47-42f02fb537a8" containerName="kserve-container"
Sep 09 20:38:47.906867 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.906399    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc9996a-b4bc-4c6b-80e1-d6ff34b6fc83" containerName="kube-rbac-proxy"
Sep 09 20:38:47.909592 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.909574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:47.912034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912007    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 09 20:38:47.912161 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912038    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:38:47.912161 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912106    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:38:47.912266 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912175    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:38:47.912759 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912741    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 09 20:38:47.912759 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.912752    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:38:47.919321 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:47.919298    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:38:48.052152 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.052114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4t52\" (UniqueName: \"kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.052349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.052159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.052349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.052229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.052349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.052307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.153394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.153360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.153571 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.153413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4t52\" (UniqueName: \"kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.153571 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.153436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.153571 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.153473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.153797 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:38:48.153594    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 09 20:38:48.153797 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:38:48.153655    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls podName:bfccbebb-2b23-4f0b-b464-288fed024123 nodeName:}" failed. No retries permitted until 2026-09-09 20:38:48.65363336 +0000 UTC m=+2618.054310530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls") pod "isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" (UID: "bfccbebb-2b23-4f0b-b464-288fed024123") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 09 20:38:48.153927 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.153909    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.154065 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.154046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.162479 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.162431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4t52\" (UniqueName: \"kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.657948 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.657904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.660338 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.660316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-h7h4w\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:48.820774 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:48.820733    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:49.092883 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:49.092855    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:38:49.095457 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:38:49.095431    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfccbebb_2b23_4f0b_b464_288fed024123.slice/crio-82a6e36d06301fc9a4de35a2d10e5d73ddee1053ab29eecabee8771d1924bfca WatchSource:0}: Error finding container 82a6e36d06301fc9a4de35a2d10e5d73ddee1053ab29eecabee8771d1924bfca: Status 404 returned error can't find the container with id 82a6e36d06301fc9a4de35a2d10e5d73ddee1053ab29eecabee8771d1924bfca
Sep 09 20:38:49.469692 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:49.469652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerStarted","Data":"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"}
Sep 09 20:38:49.469853 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:49.469728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerStarted","Data":"82a6e36d06301fc9a4de35a2d10e5d73ddee1053ab29eecabee8771d1924bfca"}
Sep 09 20:38:50.476207 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:50.476169    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfccbebb-2b23-4f0b-b464-288fed024123" containerID="8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c" exitCode=0
Sep 09 20:38:50.476595 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:50.476241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerDied","Data":"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"}
Sep 09 20:38:51.483404 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:51.483365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerStarted","Data":"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"}
Sep 09 20:38:51.483818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:51.483435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerStarted","Data":"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"}
Sep 09 20:38:51.483818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:51.483467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:51.502622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:51.502568    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podStartSLOduration=3.232012418 podStartE2EDuration="4.502548763s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.270536345s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:51.501714168 +0000 UTC m=+2620.902391359" watchObservedRunningTime="2026-09-09 20:38:51.502548763 +0000 UTC m=+2620.903225957"
Sep 09 20:38:52.488040 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:52.488004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:52.489330 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:52.489299    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:38:53.492105 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:53.492063    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:38:58.497128 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:58.497098    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:38:58.497572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:38:58.497546    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:00.138012 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.137974    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:00.141561 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.141544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.143862 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.143831    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:39:00.143982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.143885    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:39:00.144039 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.143979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 09 20:39:00.144123 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.144108    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:39:00.153250 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.153227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:00.252724 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.252689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.252895 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.252755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.252895 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.252785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.252895 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.252814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.252895 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.252873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmr5q\" (UniqueName: \"kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354249 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmr5q\" (UniqueName: \"kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.354832 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.354811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.355087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.355062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.355210 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.355133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.356819 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.356794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.362102 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.362085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmr5q\" (UniqueName: \"kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.453916 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.453887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:00.727622 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:00.727590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:00.730358 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:39:00.730331    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaae7aa_53ce_4cac_a4b8_af866446ebf2.slice/crio-4aef18ca3ac96158d8a175920070ec8a052966c022f03f9f176114dc0c9d42c6 WatchSource:0}: Error finding container 4aef18ca3ac96158d8a175920070ec8a052966c022f03f9f176114dc0c9d42c6: Status 404 returned error can't find the container with id 4aef18ca3ac96158d8a175920070ec8a052966c022f03f9f176114dc0c9d42c6
Sep 09 20:39:01.530986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:01.530950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerStarted","Data":"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"}
Sep 09 20:39:01.531399 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:01.531014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerStarted","Data":"4aef18ca3ac96158d8a175920070ec8a052966c022f03f9f176114dc0c9d42c6"}
Sep 09 20:39:05.552974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:05.552937    2582 generic.go:356] "Generic (PLEG): container finished" podID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerID="5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10" exitCode=0
Sep 09 20:39:05.553349 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:05.553012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerDied","Data":"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"}
Sep 09 20:39:06.559259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:06.559223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerStarted","Data":"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"}
Sep 09 20:39:06.559776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:06.559285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerStarted","Data":"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"}
Sep 09 20:39:06.559776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:06.559474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:06.559776 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:06.559515    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:06.581531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:06.581459    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" podStartSLOduration=2.639396264 podStartE2EDuration="6.581438333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942042069s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:06.579169462 +0000 UTC m=+2635.979846661" watchObservedRunningTime="2026-09-09 20:39:06.581438333 +0000 UTC m=+2635.982115526"
Sep 09 20:39:08.498199 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:08.498145    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:12.568543 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:12.568508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:18.498254 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:18.498211    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:28.498116 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:28.498078    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:38.498255 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:38.498217    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:42.573097 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:42.573061    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:48.498327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:48.498281    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:39:50.196006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:50.195971    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:50.196467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:50.196205    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kserve-container" containerID="cri-o://a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b" gracePeriod=30
Sep 09 20:39:50.196467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:50.196245    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kube-rbac-proxy" containerID="cri-o://96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122" gracePeriod=30
Sep 09 20:39:50.763488 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:50.763454    2582 generic.go:356] "Generic (PLEG): container finished" podID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerID="96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122" exitCode=2
Sep 09 20:39:50.763659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:50.763531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerDied","Data":"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"}
Sep 09 20:39:51.148544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.148523    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:51.271033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.270942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location\") pod \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") "
Sep 09 20:39:51.271033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271017    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config\") pod \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") "
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert\") pod \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") "
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271102    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmr5q\" (UniqueName: \"kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q\") pod \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") "
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271159    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls\") pod \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\" (UID: \"4eaae7aa-53ce-4cac-a4b8-af866446ebf2\") "
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4eaae7aa-53ce-4cac-a4b8-af866446ebf2" (UID: "4eaae7aa-53ce-4cac-a4b8-af866446ebf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271416    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4eaae7aa-53ce-4cac-a4b8-af866446ebf2" (UID: "4eaae7aa-53ce-4cac-a4b8-af866446ebf2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:39:51.271550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.271439    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4eaae7aa-53ce-4cac-a4b8-af866446ebf2" (UID: "4eaae7aa-53ce-4cac-a4b8-af866446ebf2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:39:51.273320 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.273296    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q" (OuterVolumeSpecName: "kube-api-access-rmr5q") pod "4eaae7aa-53ce-4cac-a4b8-af866446ebf2" (UID: "4eaae7aa-53ce-4cac-a4b8-af866446ebf2"). InnerVolumeSpecName "kube-api-access-rmr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:51.273394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.273335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4eaae7aa-53ce-4cac-a4b8-af866446ebf2" (UID: "4eaae7aa-53ce-4cac-a4b8-af866446ebf2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:51.372476 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.372442    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:39:51.372476 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.372473    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:39:51.372476 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.372483    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:39:51.372726 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.372491    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmr5q\" (UniqueName: \"kubernetes.io/projected/4eaae7aa-53ce-4cac-a4b8-af866446ebf2-kube-api-access-rmr5q\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:39:51.770487 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.770450    2582 generic.go:356] "Generic (PLEG): container finished" podID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerID="a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b" exitCode=0
Sep 09 20:39:51.770659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.770501    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"
Sep 09 20:39:51.770659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.770532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerDied","Data":"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"}
Sep 09 20:39:51.770659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.770594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl" event={"ID":"4eaae7aa-53ce-4cac-a4b8-af866446ebf2","Type":"ContainerDied","Data":"4aef18ca3ac96158d8a175920070ec8a052966c022f03f9f176114dc0c9d42c6"}
Sep 09 20:39:51.770659 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.770623    2582 scope.go:117] "RemoveContainer" containerID="96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"
Sep 09 20:39:51.781039 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.781013    2582 scope.go:117] "RemoveContainer" containerID="a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"
Sep 09 20:39:51.789237 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.789197    2582 scope.go:117] "RemoveContainer" containerID="5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"
Sep 09 20:39:51.795076 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.795052    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:51.799072 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799050    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-46phl"]
Sep 09 20:39:51.799164 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799105    2582 scope.go:117] "RemoveContainer" containerID="96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"
Sep 09 20:39:51.799441 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:39:51.799416    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122\": container with ID starting with 96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122 not found: ID does not exist" containerID="96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"
Sep 09 20:39:51.799556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799450    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122"} err="failed to get container status \"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122\": rpc error: code = NotFound desc = could not find container \"96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122\": container with ID starting with 96ed813d9ff051f7584e7cbace8fd784eb4e8dd8fda58c1efc7045c76e39d122 not found: ID does not exist"
Sep 09 20:39:51.799556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799476    2582 scope.go:117] "RemoveContainer" containerID="a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"
Sep 09 20:39:51.799816 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:39:51.799793    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b\": container with ID starting with a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b not found: ID does not exist" containerID="a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"
Sep 09 20:39:51.799911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799820    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b"} err="failed to get container status \"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b\": rpc error: code = NotFound desc = could not find container \"a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b\": container with ID starting with a0c2ffdd42bebff6fe74019a767080549b32adf382720b5767afc65349b74a4b not found: ID does not exist"
Sep 09 20:39:51.799911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.799837    2582 scope.go:117] "RemoveContainer" containerID="5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"
Sep 09 20:39:51.800155 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:39:51.800127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10\": container with ID starting with 5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10 not found: ID does not exist" containerID="5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"
Sep 09 20:39:51.800242 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:51.800162    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10"} err="failed to get container status \"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10\": rpc error: code = NotFound desc = could not find container \"5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10\": container with ID starting with 5ef445a24948cc9ab6cee099d78268a7049fe91889dee5e85f23d5b89e745d10 not found: ID does not exist"
Sep 09 20:39:53.124080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:53.124037    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" path="/var/lib/kubelet/pods/4eaae7aa-53ce-4cac-a4b8-af866446ebf2/volumes"
Sep 09 20:39:55.377027 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.376989    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:39:55.377628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="storage-initializer"
Sep 09 20:39:55.377628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377629    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="storage-initializer"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377648    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kube-rbac-proxy"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377658    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kube-rbac-proxy"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377698    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kserve-container"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377710    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kserve-container"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377799    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kserve-container"
Sep 09 20:39:55.377843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.377816    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eaae7aa-53ce-4cac-a4b8-af866446ebf2" containerName="kube-rbac-proxy"
Sep 09 20:39:55.381606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.381585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.383848 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.383822    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:39:55.383971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.383852    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 20:39:55.383971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.383896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 09 20:39:55.383971 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.383902    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:39:55.388461 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.388437    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:39:55.509846 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.509813    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.510016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.509869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.510016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.509951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.510016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.509981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.510016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.510013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-242md\" (UniqueName: \"kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.610768 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.610727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.610961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.610780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.610961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.610823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.610961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.610841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.610961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.610874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-242md\" (UniqueName: \"kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.611206 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:39:55.610968    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 09 20:39:55.611206 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:39:55.611040    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls podName:38bfc6d7-bff4-4410-914b-0d7a670b8f44 nodeName:}" failed. No retries permitted until 2026-09-09 20:39:56.111018061 +0000 UTC m=+2685.511695244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 09 20:39:55.611306 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.611207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.611503 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.611485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.611544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.611533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:55.619135 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:55.619112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-242md\" (UniqueName: \"kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:56.116117 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.116074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:56.118466 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.118439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:56.292756 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.292723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:39:56.566072 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.566034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:39:56.569278 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:39:56.569248    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bfc6d7_bff4_4410_914b_0d7a670b8f44.slice/crio-427fafd105d0c8222486b18d188cd0f88d2f3739723673e9cda95b844373d9a6 WatchSource:0}: Error finding container 427fafd105d0c8222486b18d188cd0f88d2f3739723673e9cda95b844373d9a6: Status 404 returned error can't find the container with id 427fafd105d0c8222486b18d188cd0f88d2f3739723673e9cda95b844373d9a6
Sep 09 20:39:56.802259 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.802220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerStarted","Data":"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"}
Sep 09 20:39:56.802448 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:56.802283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerStarted","Data":"427fafd105d0c8222486b18d188cd0f88d2f3739723673e9cda95b844373d9a6"}
Sep 09 20:39:58.498101 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:39:58.498060    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:40:00.828104 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:00.828068    2582 generic.go:356] "Generic (PLEG): container finished" podID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerID="ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7" exitCode=0
Sep 09 20:40:00.828483 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:00.828141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerDied","Data":"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"}
Sep 09 20:40:01.834493 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:01.834457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerStarted","Data":"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"}
Sep 09 20:40:01.835006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:01.834520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerStarted","Data":"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"}
Sep 09 20:40:01.835006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:01.834714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:01.884608 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:01.884551    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" podStartSLOduration=3.145372474 podStartE2EDuration="6.884534237s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739161763s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:01.884250716 +0000 UTC m=+2691.284927907" watchObservedRunningTime="2026-09-09 20:40:01.884534237 +0000 UTC m=+2691.285211479"
Sep 09 20:40:02.839601 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:02.839560    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:02.840808 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:02.840777    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 09 20:40:03.844570 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:03.844471    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 09 20:40:08.498877 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:08.498843    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:40:08.849420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:08.849325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:08.850394 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:08.850358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:15.425618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:15.425584    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:40:15.426114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:15.425832    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container" containerID="cri-o://786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6" gracePeriod=30
Sep 09 20:40:15.426114 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:15.425875    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kube-rbac-proxy" containerID="cri-o://4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43" gracePeriod=30
Sep 09 20:40:15.890514 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:15.890481    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bfc6d7_bff4_4410_914b_0d7a670b8f44.slice/crio-786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 20:40:15.901734 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:15.901707    2582 generic.go:356] "Generic (PLEG): container finished" podID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerID="4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43" exitCode=2
Sep 09 20:40:15.901865 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:15.901754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerDied","Data":"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"}
Sep 09 20:40:16.077180 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.077153    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:16.186986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.186897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert\") pod \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") "
Sep 09 20:40:16.186986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.186935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config\") pod \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") "
Sep 09 20:40:16.186986 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.186956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") pod \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") "
Sep 09 20:40:16.187274 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.187028    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-242md\" (UniqueName: \"kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md\") pod \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") "
Sep 09 20:40:16.187274 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.187051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location\") pod \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\" (UID: \"38bfc6d7-bff4-4410-914b-0d7a670b8f44\") "
Sep 09 20:40:16.187385 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.187349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38bfc6d7-bff4-4410-914b-0d7a670b8f44" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:16.187467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.187438    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38bfc6d7-bff4-4410-914b-0d7a670b8f44" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:16.187520 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.187443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38bfc6d7-bff4-4410-914b-0d7a670b8f44" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:16.189119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.189085    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md" (OuterVolumeSpecName: "kube-api-access-242md") pod "38bfc6d7-bff4-4410-914b-0d7a670b8f44" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44"). InnerVolumeSpecName "kube-api-access-242md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:16.189224 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.189181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38bfc6d7-bff4-4410-914b-0d7a670b8f44" (UID: "38bfc6d7-bff4-4410-914b-0d7a670b8f44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:16.287880 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.287847    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:16.287880 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.287872    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:16.287880 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.287882    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38bfc6d7-bff4-4410-914b-0d7a670b8f44-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:16.288099 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.287892    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-242md\" (UniqueName: \"kubernetes.io/projected/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kube-api-access-242md\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:16.288099 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.287900    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38bfc6d7-bff4-4410-914b-0d7a670b8f44-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:16.908300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.908266    2582 generic.go:356] "Generic (PLEG): container finished" podID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerID="786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6" exitCode=0
Sep 09 20:40:16.908300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.908302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerDied","Data":"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"}
Sep 09 20:40:16.908830 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.908320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"
Sep 09 20:40:16.908830 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.908349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs" event={"ID":"38bfc6d7-bff4-4410-914b-0d7a670b8f44","Type":"ContainerDied","Data":"427fafd105d0c8222486b18d188cd0f88d2f3739723673e9cda95b844373d9a6"}
Sep 09 20:40:16.908830 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.908375    2582 scope.go:117] "RemoveContainer" containerID="4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"
Sep 09 20:40:16.924376 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.924354    2582 scope.go:117] "RemoveContainer" containerID="786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"
Sep 09 20:40:16.932070 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.932046    2582 scope.go:117] "RemoveContainer" containerID="ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"
Sep 09 20:40:16.934339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.934313    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:40:16.941338 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.941312    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-zztzs"]
Sep 09 20:40:16.941595 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.941581    2582 scope.go:117] "RemoveContainer" containerID="4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"
Sep 09 20:40:16.941876 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:16.941855    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43\": container with ID starting with 4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43 not found: ID does not exist" containerID="4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"
Sep 09 20:40:16.941949 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.941887    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43"} err="failed to get container status \"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43\": rpc error: code = NotFound desc = could not find container \"4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43\": container with ID starting with 4f82db44fb2fba501af51d3e1b591b707ce431da209eea7b3b8841fc26f69c43 not found: ID does not exist"
Sep 09 20:40:16.941949 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.941906    2582 scope.go:117] "RemoveContainer" containerID="786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"
Sep 09 20:40:16.942149 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:16.942132    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6\": container with ID starting with 786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6 not found: ID does not exist" containerID="786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"
Sep 09 20:40:16.942189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.942154    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6"} err="failed to get container status \"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6\": rpc error: code = NotFound desc = could not find container \"786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6\": container with ID starting with 786af0d4dcde804e3f8220195e1cc2196eabda3a14e0bafc62417cb8681fc0e6 not found: ID does not exist"
Sep 09 20:40:16.942189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.942168    2582 scope.go:117] "RemoveContainer" containerID="ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"
Sep 09 20:40:16.942394 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:16.942377    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7\": container with ID starting with ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7 not found: ID does not exist" containerID="ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"
Sep 09 20:40:16.942444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:16.942397    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7"} err="failed to get container status \"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7\": rpc error: code = NotFound desc = could not find container \"ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7\": container with ID starting with ba2b8b24846aab95b82803aa17d611b69ae8c27292f5b1c4bafb7d5b57d8b8a7 not found: ID does not exist"
Sep 09 20:40:17.125111 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:17.125075    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" path="/var/lib/kubelet/pods/38bfc6d7-bff4-4410-914b-0d7a670b8f44/volumes"
Sep 09 20:40:18.012081 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.012042    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:40:18.012506 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.012284    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" containerID="cri-o://6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36" gracePeriod=30
Sep 09 20:40:18.012506 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.012333    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kube-rbac-proxy" containerID="cri-o://a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7" gracePeriod=30
Sep 09 20:40:18.492406 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.492369    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 09 20:40:18.497522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.497494    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 09 20:40:18.921440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.921348    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfccbebb-2b23-4f0b-b464-288fed024123" containerID="a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7" exitCode=2
Sep 09 20:40:18.921440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:18.921422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerDied","Data":"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"}
Sep 09 20:40:20.638350 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638311    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638656    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="storage-initializer"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638667    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="storage-initializer"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638719    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kube-rbac-proxy"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638725    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kube-rbac-proxy"
Sep 09 20:40:20.638779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kserve-container"
Sep 09 20:40:20.639049 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.638789    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38bfc6d7-bff4-4410-914b-0d7a670b8f44" containerName="kube-rbac-proxy"
Sep 09 20:40:20.642011 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.641995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.644750 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.644720    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:40:20.644894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.644758    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:40:20.644894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.644803    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:40:20.645111 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.645087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 09 20:40:20.657862 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.657838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:40:20.725783 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.725755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.725940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.725824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmhp\" (UniqueName: \"kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.725940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.725868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.725940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.725896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.726060 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.725948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827101 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmhp\" (UniqueName: \"kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827284 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827284 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827284 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827411 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827627 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.827996 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.828061 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.827976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.829724 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.829707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.837972 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.837951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmhp\" (UniqueName: \"kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:20.952994 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:20.952965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:21.231046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:21.230971    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:40:21.233461 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:40:21.233431    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cd4be3_2e95_4dd2_9517_185e7ce9b947.slice/crio-dac5ac9fc3b8422a57d7b7015b869ed83f3376bbd751777156eb2d087278acf4 WatchSource:0}: Error finding container dac5ac9fc3b8422a57d7b7015b869ed83f3376bbd751777156eb2d087278acf4: Status 404 returned error can't find the container with id dac5ac9fc3b8422a57d7b7015b869ed83f3376bbd751777156eb2d087278acf4
Sep 09 20:40:21.936858 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:21.936819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerStarted","Data":"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"}
Sep 09 20:40:21.937239 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:21.936891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerStarted","Data":"dac5ac9fc3b8422a57d7b7015b869ed83f3376bbd751777156eb2d087278acf4"}
Sep 09 20:40:22.751318 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.751294    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:40:22.843432 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config\") pod \"bfccbebb-2b23-4f0b-b464-288fed024123\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") "
Sep 09 20:40:22.843591 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4t52\" (UniqueName: \"kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52\") pod \"bfccbebb-2b23-4f0b-b464-288fed024123\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") "
Sep 09 20:40:22.843591 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location\") pod \"bfccbebb-2b23-4f0b-b464-288fed024123\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") "
Sep 09 20:40:22.843591 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") pod \"bfccbebb-2b23-4f0b-b464-288fed024123\" (UID: \"bfccbebb-2b23-4f0b-b464-288fed024123\") "
Sep 09 20:40:22.843869 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfccbebb-2b23-4f0b-b464-288fed024123" (UID: "bfccbebb-2b23-4f0b-b464-288fed024123"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:22.843869 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.843823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfccbebb-2b23-4f0b-b464-288fed024123" (UID: "bfccbebb-2b23-4f0b-b464-288fed024123"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:22.845532 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.845503    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52" (OuterVolumeSpecName: "kube-api-access-j4t52") pod "bfccbebb-2b23-4f0b-b464-288fed024123" (UID: "bfccbebb-2b23-4f0b-b464-288fed024123"). InnerVolumeSpecName "kube-api-access-j4t52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:22.845635 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.845563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfccbebb-2b23-4f0b-b464-288fed024123" (UID: "bfccbebb-2b23-4f0b-b464-288fed024123"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:22.943101 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.943068    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfccbebb-2b23-4f0b-b464-288fed024123" containerID="6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36" exitCode=0
Sep 09 20:40:22.943522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.943125    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"
Sep 09 20:40:22.943522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.943160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerDied","Data":"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"}
Sep 09 20:40:22.943522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.943260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w" event={"ID":"bfccbebb-2b23-4f0b-b464-288fed024123","Type":"ContainerDied","Data":"82a6e36d06301fc9a4de35a2d10e5d73ddee1053ab29eecabee8771d1924bfca"}
Sep 09 20:40:22.943522 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.943290    2582 scope.go:117] "RemoveContainer" containerID="a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"
Sep 09 20:40:22.944033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.944005    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfccbebb-2b23-4f0b-b464-288fed024123-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:22.944033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.944030    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfccbebb-2b23-4f0b-b464-288fed024123-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:22.944192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.944041    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfccbebb-2b23-4f0b-b464-288fed024123-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:22.944192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.944050    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4t52\" (UniqueName: \"kubernetes.io/projected/bfccbebb-2b23-4f0b-b464-288fed024123-kube-api-access-j4t52\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:40:22.952244 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.952227    2582 scope.go:117] "RemoveContainer" containerID="6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"
Sep 09 20:40:22.959533 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.959515    2582 scope.go:117] "RemoveContainer" containerID="8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"
Sep 09 20:40:22.965307 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.965285    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:40:22.967196 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.967182    2582 scope.go:117] "RemoveContainer" containerID="a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"
Sep 09 20:40:22.967438 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:22.967422    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7\": container with ID starting with a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7 not found: ID does not exist" containerID="a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"
Sep 09 20:40:22.967491 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.967445    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7"} err="failed to get container status \"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7\": rpc error: code = NotFound desc = could not find container \"a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7\": container with ID starting with a6e97f6c2b3004501b559cf78bbe9a5d347afebb59c3d6049e6b4f2f300aa2f7 not found: ID does not exist"
Sep 09 20:40:22.967491 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.967459    2582 scope.go:117] "RemoveContainer" containerID="6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"
Sep 09 20:40:22.967784 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:22.967767    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36\": container with ID starting with 6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36 not found: ID does not exist" containerID="6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"
Sep 09 20:40:22.967831 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.967789    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36"} err="failed to get container status \"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36\": rpc error: code = NotFound desc = could not find container \"6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36\": container with ID starting with 6a172df46f4cba299ff7bcb8044657d785b0c732ad66ce343f84d660145c9e36 not found: ID does not exist"
Sep 09 20:40:22.967831 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.967803    2582 scope.go:117] "RemoveContainer" containerID="8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"
Sep 09 20:40:22.968067 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:40:22.968040    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c\": container with ID starting with 8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c not found: ID does not exist" containerID="8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"
Sep 09 20:40:22.968161 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.968073    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c"} err="failed to get container status \"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c\": rpc error: code = NotFound desc = could not find container \"8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c\": container with ID starting with 8f044655b9304ac60c8b3d69d8781aa878f3ef242a745fcd5c1879034c07984c not found: ID does not exist"
Sep 09 20:40:22.970577 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:22.970557    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-h7h4w"]
Sep 09 20:40:23.124741 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:23.124650    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" path="/var/lib/kubelet/pods/bfccbebb-2b23-4f0b-b464-288fed024123/volumes"
Sep 09 20:40:25.960485 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:25.960453    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerID="e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a" exitCode=0
Sep 09 20:40:25.960891 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:25.960525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerDied","Data":"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"}
Sep 09 20:40:26.968538 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:26.968503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerStarted","Data":"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"}
Sep 09 20:40:26.968936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:26.968567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerStarted","Data":"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"}
Sep 09 20:40:26.968936 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:26.968753    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:26.990785 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:26.990735    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" podStartSLOduration=3.120380058 podStartE2EDuration="6.990717298s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.87033724s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:26.987392022 +0000 UTC m=+2716.388069213" watchObservedRunningTime="2026-09-09 20:40:26.990717298 +0000 UTC m=+2716.391394489"
Sep 09 20:40:27.972742 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:27.972707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:40:33.982826 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:40:33.982798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:41:03.987109 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:03.987071    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:41:10.662865 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:10.662831    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:41:10.663291 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:10.663064    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kserve-container" containerID="cri-o://66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc" gracePeriod=30
Sep 09 20:41:10.663291 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:10.663093    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kube-rbac-proxy" containerID="cri-o://ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839" gracePeriod=30
Sep 09 20:41:11.179690 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:11.179642    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerID="ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839" exitCode=2
Sep 09 20:41:11.179855 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:11.179712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerDied","Data":"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"}
Sep 09 20:41:13.978208 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:13.978163    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 09 20:41:14.776917 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:14.776869    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v2/models/isvc-mlflow-v2-runtime/ready\": read tcp 10.134.0.2:46500->10.134.0.59:8080: read: connection reset by peer"
Sep 09 20:41:15.405876 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.405854    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:41:15.495527 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxmhp\" (UniqueName: \"kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp\") pod \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") "
Sep 09 20:41:15.495527 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config\") pod \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") "
Sep 09 20:41:15.495787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert\") pod \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") "
Sep 09 20:41:15.495787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495579    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location\") pod \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") "
Sep 09 20:41:15.495787 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495613    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls\") pod \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\" (UID: \"b1cd4be3-2e95-4dd2-9517-185e7ce9b947\") "
Sep 09 20:41:15.495991 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1cd4be3-2e95-4dd2-9517-185e7ce9b947" (UID: "b1cd4be3-2e95-4dd2-9517-185e7ce9b947"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:15.496035 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495970    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1cd4be3-2e95-4dd2-9517-185e7ce9b947" (UID: "b1cd4be3-2e95-4dd2-9517-185e7ce9b947"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:15.496035 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.495992    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1cd4be3-2e95-4dd2-9517-185e7ce9b947" (UID: "b1cd4be3-2e95-4dd2-9517-185e7ce9b947"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:15.497618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.497598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1cd4be3-2e95-4dd2-9517-185e7ce9b947" (UID: "b1cd4be3-2e95-4dd2-9517-185e7ce9b947"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:15.497743 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.497642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp" (OuterVolumeSpecName: "kube-api-access-jxmhp") pod "b1cd4be3-2e95-4dd2-9517-185e7ce9b947" (UID: "b1cd4be3-2e95-4dd2-9517-185e7ce9b947"). InnerVolumeSpecName "kube-api-access-jxmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:15.596228 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.596200    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:41:15.596228 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.596225    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:41:15.596407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.596236    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:41:15.596407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.596247    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:41:15.596407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.596257    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmhp\" (UniqueName: \"kubernetes.io/projected/b1cd4be3-2e95-4dd2-9517-185e7ce9b947-kube-api-access-jxmhp\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:41:15.872920 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.872811    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:41:15.873749 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873698    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kube-rbac-proxy"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873754    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kube-rbac-proxy"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873782    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873791    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873816    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kube-rbac-proxy"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873824    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kube-rbac-proxy"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873854    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="storage-initializer"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873862    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="storage-initializer"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873877    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kserve-container"
Sep 09 20:41:15.873924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873885    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kserve-container"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.873983    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="storage-initializer"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.874001    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="storage-initializer"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.874264    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kserve-container"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.874286    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerName="kube-rbac-proxy"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.874296    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kube-rbac-proxy"
Sep 09 20:41:15.874327 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.874306    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfccbebb-2b23-4f0b-b464-288fed024123" containerName="kserve-container"
Sep 09 20:41:15.878544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.878521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:15.880938 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.880919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 09 20:41:15.881205 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.881185    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 09 20:41:15.886562 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.886531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:41:15.898983 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.898917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:15.899097 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.899017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:15.899097 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.899047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:15.899189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.899111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m5p\" (UniqueName: \"kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:15.899189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:15.899169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000301 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m5p\" (UniqueName: \"kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.000843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.000821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.001042 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.001018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.001119 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.001075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.003106 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.003087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.008887 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.008866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m5p\" (UniqueName: \"kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.191391 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.191300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:16.203437 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.203408    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" containerID="66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc" exitCode=0
Sep 09 20:41:16.203557 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.203484    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"
Sep 09 20:41:16.203557 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.203482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerDied","Data":"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"}
Sep 09 20:41:16.203638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.203566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj" event={"ID":"b1cd4be3-2e95-4dd2-9517-185e7ce9b947","Type":"ContainerDied","Data":"dac5ac9fc3b8422a57d7b7015b869ed83f3376bbd751777156eb2d087278acf4"}
Sep 09 20:41:16.203638 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.203601    2582 scope.go:117] "RemoveContainer" containerID="ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"
Sep 09 20:41:16.211713 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.211692    2582 scope.go:117] "RemoveContainer" containerID="66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"
Sep 09 20:41:16.221144 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.221122    2582 scope.go:117] "RemoveContainer" containerID="e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"
Sep 09 20:41:16.229958 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.229921    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:41:16.233900 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.233861    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vwvpj"]
Sep 09 20:41:16.233982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.233948    2582 scope.go:117] "RemoveContainer" containerID="ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"
Sep 09 20:41:16.234349 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:41:16.234328    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839\": container with ID starting with ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839 not found: ID does not exist" containerID="ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"
Sep 09 20:41:16.234406 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.234358    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839"} err="failed to get container status \"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839\": rpc error: code = NotFound desc = could not find container \"ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839\": container with ID starting with ecd0cc4d790c857895c64b1b3270691f0ed64b83fb6bb5863c468e94483d0839 not found: ID does not exist"
Sep 09 20:41:16.234406 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.234377    2582 scope.go:117] "RemoveContainer" containerID="66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"
Sep 09 20:41:16.234694 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:41:16.234656    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc\": container with ID starting with 66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc not found: ID does not exist" containerID="66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"
Sep 09 20:41:16.234755 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.234710    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc"} err="failed to get container status \"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc\": rpc error: code = NotFound desc = could not find container \"66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc\": container with ID starting with 66cbbc3fa73c7c2ab8dfc27fe9d37a3d815f6b7e0eea59398ef232c917f601cc not found: ID does not exist"
Sep 09 20:41:16.234755 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.234727    2582 scope.go:117] "RemoveContainer" containerID="e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"
Sep 09 20:41:16.234973 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:41:16.234954    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a\": container with ID starting with e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a not found: ID does not exist" containerID="e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"
Sep 09 20:41:16.235033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.234975    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a"} err="failed to get container status \"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a\": rpc error: code = NotFound desc = could not find container \"e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a\": container with ID starting with e2e50f390fe9c0fade72a7f9719b7f45f802380e14f9ed7af8fe8a71e26bdb9a not found: ID does not exist"
Sep 09 20:41:16.470495 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:16.469394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:41:16.473289 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:41:16.473262    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7dc00d1_e950_4985_bbc9_85c9c2b21bb9.slice/crio-b0ed57cbb6ea8c856a3d2358716493f2deb8d7134432f9af5918b335fb7cf145 WatchSource:0}: Error finding container b0ed57cbb6ea8c856a3d2358716493f2deb8d7134432f9af5918b335fb7cf145: Status 404 returned error can't find the container with id b0ed57cbb6ea8c856a3d2358716493f2deb8d7134432f9af5918b335fb7cf145
Sep 09 20:41:17.124707 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:17.124661    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1cd4be3-2e95-4dd2-9517-185e7ce9b947" path="/var/lib/kubelet/pods/b1cd4be3-2e95-4dd2-9517-185e7ce9b947/volumes"
Sep 09 20:41:17.209156 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:17.209123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerStarted","Data":"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"}
Sep 09 20:41:17.209363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:17.209193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerStarted","Data":"b0ed57cbb6ea8c856a3d2358716493f2deb8d7134432f9af5918b335fb7cf145"}
Sep 09 20:41:20.225569 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:20.225536    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerID="83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7" exitCode=0
Sep 09 20:41:20.225996 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:20.225605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerDied","Data":"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"}
Sep 09 20:41:20.294061 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:20.294028    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:41:21.234998 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:21.234924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerStarted","Data":"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"}
Sep 09 20:41:23.246644 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:23.246601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerStarted","Data":"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"}
Sep 09 20:41:23.247034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:23.246695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerStarted","Data":"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"}
Sep 09 20:41:23.247034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:23.246731    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:23.247034 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:23.246763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:23.268551 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:23.268501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podStartSLOduration=1.9615768249999999 podStartE2EDuration="8.268485939s" totalImagesPullingTime="2.751980617s" totalInitContainerRuntime="3.554928497s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:23.266446676 +0000 UTC m=+2772.667123871" watchObservedRunningTime="2026-09-09 20:41:23.268485939 +0000 UTC m=+2772.669163131"
Sep 09 20:41:24.252572 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:24.252542    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:30.262082 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:30.262050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:41:50.263293 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:41:50.263255    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 09 20:42:00.264046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:00.264011    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:42:30.264730 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:30.264700    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:42:35.890331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:35.890292    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:42:35.890914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:35.890544    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container" containerID="cri-o://33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d" gracePeriod=30
Sep 09 20:42:35.890914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:35.890639    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-agent" containerID="cri-o://cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0" gracePeriod=30
Sep 09 20:42:35.890914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:35.890573    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" containerID="cri-o://630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636" gracePeriod=30
Sep 09 20:42:36.587080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:36.587043    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerID="630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636" exitCode=2
Sep 09 20:42:36.587261 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:36.587084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerDied","Data":"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"}
Sep 09 20:42:38.599098 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:38.599067    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerID="33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d" exitCode=0
Sep 09 20:42:38.599477 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:38.599141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerDied","Data":"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"}
Sep 09 20:42:40.257770 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:40.257727    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:42:40.262919 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:40.262890    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 09 20:42:45.257405 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:45.257361    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:42:50.257407 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:50.257362    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:42:50.257815 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:50.257509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:42:50.262643 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:50.262618    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 09 20:42:55.257824 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:42:55.257782    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:43:00.256920 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:00.256877    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:43:00.262790 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:00.262764    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 09 20:43:00.262888 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:00.262875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:43:05.257465 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:05.257382    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 09 20:43:06.071884 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.071860    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:43:06.226585 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.226555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config\") pod \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") "
Sep 09 20:43:06.226792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.226628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26m5p\" (UniqueName: \"kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p\") pod \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") "
Sep 09 20:43:06.226792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.226658    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls\") pod \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") "
Sep 09 20:43:06.226792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.226719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert\") pod \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") "
Sep 09 20:43:06.226792 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.226737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location\") pod \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\" (UID: \"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9\") "
Sep 09 20:43:06.227074 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.227035    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" (UID: "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:06.227194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.227113    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" (UID: "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:06.227194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.227144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" (UID: "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:06.228835 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.228811    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" (UID: "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:06.228926 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.228910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p" (OuterVolumeSpecName: "kube-api-access-26m5p") pod "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" (UID: "b7dc00d1-e950-4985-bbc9-85c9c2b21bb9"). InnerVolumeSpecName "kube-api-access-26m5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:06.328113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.328077    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26m5p\" (UniqueName: \"kubernetes.io/projected/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-api-access-26m5p\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:06.328113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.328107    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:06.328113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.328118    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:06.328549 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.328126    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:06.328549 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.328134    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:06.733200 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.733168    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerID="cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0" exitCode=137
Sep 09 20:43:06.733340 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.733237    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"
Sep 09 20:43:06.733340 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.733253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerDied","Data":"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"}
Sep 09 20:43:06.733340 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.733318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb" event={"ID":"b7dc00d1-e950-4985-bbc9-85c9c2b21bb9","Type":"ContainerDied","Data":"b0ed57cbb6ea8c856a3d2358716493f2deb8d7134432f9af5918b335fb7cf145"}
Sep 09 20:43:06.733458 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.733351    2582 scope.go:117] "RemoveContainer" containerID="630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"
Sep 09 20:43:06.744107 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.744089    2582 scope.go:117] "RemoveContainer" containerID="cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"
Sep 09 20:43:06.751647 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.751628    2582 scope.go:117] "RemoveContainer" containerID="33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"
Sep 09 20:43:06.759010 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.758987    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:43:06.759603 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.759581    2582 scope.go:117] "RemoveContainer" containerID="83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"
Sep 09 20:43:06.765978 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.765955    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-24wdb"]
Sep 09 20:43:06.767664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.767646    2582 scope.go:117] "RemoveContainer" containerID="630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"
Sep 09 20:43:06.767974 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:06.767954    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636\": container with ID starting with 630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636 not found: ID does not exist" containerID="630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"
Sep 09 20:43:06.768025 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.767987    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636"} err="failed to get container status \"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636\": rpc error: code = NotFound desc = could not find container \"630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636\": container with ID starting with 630d25cb602a42abf9bdb015c756799e54774a597108689a0cc279b070de3636 not found: ID does not exist"
Sep 09 20:43:06.768025 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768007    2582 scope.go:117] "RemoveContainer" containerID="cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"
Sep 09 20:43:06.768244 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:06.768224    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0\": container with ID starting with cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0 not found: ID does not exist" containerID="cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"
Sep 09 20:43:06.768314 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768253    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0"} err="failed to get container status \"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0\": rpc error: code = NotFound desc = could not find container \"cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0\": container with ID starting with cdecd59b1be272afdf37c9a19b9fb9ad8c74a0d5bbbee7f6082e03662333cbc0 not found: ID does not exist"
Sep 09 20:43:06.768314 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768277    2582 scope.go:117] "RemoveContainer" containerID="33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"
Sep 09 20:43:06.768486 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:06.768470    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d\": container with ID starting with 33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d not found: ID does not exist" containerID="33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"
Sep 09 20:43:06.768550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768492    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d"} err="failed to get container status \"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d\": rpc error: code = NotFound desc = could not find container \"33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d\": container with ID starting with 33ea69a0bc3dd1339e5693ae5992a7a2e7e109b4d1f0457552634e9c9a906f2d not found: ID does not exist"
Sep 09 20:43:06.768550 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768511    2582 scope.go:117] "RemoveContainer" containerID="83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"
Sep 09 20:43:06.768752 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:06.768733    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7\": container with ID starting with 83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7 not found: ID does not exist" containerID="83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"
Sep 09 20:43:06.768812 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:06.768758    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7"} err="failed to get container status \"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7\": rpc error: code = NotFound desc = could not find container \"83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7\": container with ID starting with 83bb37de5d4572be58b94a148be58256c2e44831c9dd853fca89217d288557b7 not found: ID does not exist"
Sep 09 20:43:07.125913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:07.125822    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" path="/var/lib/kubelet/pods/b7dc00d1-e950-4985-bbc9-85c9c2b21bb9/volumes"
Sep 09 20:43:21.487981 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.487949    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488288    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-agent"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488299    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-agent"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488315    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488321    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488337    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488343    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488349    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="storage-initializer"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488354    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="storage-initializer"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488411    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-container"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488419    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kserve-agent"
Sep 09 20:43:21.488436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.488425    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc00d1-e950-4985-bbc9-85c9c2b21bb9" containerName="kube-rbac-proxy"
Sep 09 20:43:21.493220 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.493201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.495607 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.495577    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:43:21.495746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.495615    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 09 20:43:21.495746 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.495669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:43:21.496440 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.496421    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:43:21.496480 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.496458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 09 20:43:21.496516 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.496494    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:43:21.500617 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.500350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:21.557515 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.557472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.557725 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.557589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.557725 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.557618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.557725 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.557640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drjl\" (UniqueName: \"kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.658435 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.658401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.658631 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.658490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.658631 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.658518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.658631 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.658570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drjl\" (UniqueName: \"kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.658937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.658915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.659174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.659153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.660982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.660955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.667039 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.667015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drjl\" (UniqueName: \"kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:21.804395 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:21.804292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:22.078818 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:22.078780    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:22.080983 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:43:22.080957    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa244737_6f97_4e2d_8c49_0e4d1c721b03.slice/crio-1e9e7b3f40590ceecd189ea8d578c7cfd7eb85e63f1db91672d0a88be0ae3585 WatchSource:0}: Error finding container 1e9e7b3f40590ceecd189ea8d578c7cfd7eb85e63f1db91672d0a88be0ae3585: Status 404 returned error can't find the container with id 1e9e7b3f40590ceecd189ea8d578c7cfd7eb85e63f1db91672d0a88be0ae3585
Sep 09 20:43:22.815087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:22.815046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerStarted","Data":"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"}
Sep 09 20:43:22.815527 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:22.815142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerStarted","Data":"1e9e7b3f40590ceecd189ea8d578c7cfd7eb85e63f1db91672d0a88be0ae3585"}
Sep 09 20:43:26.834790 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:26.834760    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/0.log"
Sep 09 20:43:26.835236 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:26.834801    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerID="0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790" exitCode=1
Sep 09 20:43:26.835236 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:26.834843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerDied","Data":"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"}
Sep 09 20:43:27.841668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:27.841638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/0.log"
Sep 09 20:43:27.842079 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:27.841739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerStarted","Data":"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"}
Sep 09 20:43:31.464062 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:31.464030    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:31.464436 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:31.464239    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer" containerID="cri-o://7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab" gracePeriod=30
Sep 09 20:43:32.210951 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.210929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/1.log"
Sep 09 20:43:32.211331 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.211315    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/0.log"
Sep 09 20:43:32.213625 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.213611    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:32.346149 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls\") pod \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") "
Sep 09 20:43:32.346149 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location\") pod \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") "
Sep 09 20:43:32.346325 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config\") pod \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") "
Sep 09 20:43:32.346325 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7drjl\" (UniqueName: \"kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl\") pod \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\" (UID: \"fa244737-6f97-4e2d-8c49-0e4d1c721b03\") "
Sep 09 20:43:32.346454 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa244737-6f97-4e2d-8c49-0e4d1c721b03" (UID: "fa244737-6f97-4e2d-8c49-0e4d1c721b03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:32.346549 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.346532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa244737-6f97-4e2d-8c49-0e4d1c721b03" (UID: "fa244737-6f97-4e2d-8c49-0e4d1c721b03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:32.348162 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.348141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa244737-6f97-4e2d-8c49-0e4d1c721b03" (UID: "fa244737-6f97-4e2d-8c49-0e4d1c721b03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:32.348260 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.348245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl" (OuterVolumeSpecName: "kube-api-access-7drjl") pod "fa244737-6f97-4e2d-8c49-0e4d1c721b03" (UID: "fa244737-6f97-4e2d-8c49-0e4d1c721b03"). InnerVolumeSpecName "kube-api-access-7drjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:32.446992 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.446938    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drjl\" (UniqueName: \"kubernetes.io/projected/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-api-access-7drjl\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:32.446992 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.446970    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa244737-6f97-4e2d-8c49-0e4d1c721b03-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:32.446992 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.446987    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:32.447255 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.446999    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa244737-6f97-4e2d-8c49-0e4d1c721b03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:43:32.588111 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:43:32.588469 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588441    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.588469 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588452    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.588541 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588475    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.588541 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588480    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.588541 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588536    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.588637 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.588544    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerName="storage-initializer"
Sep 09 20:43:32.591694 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.591668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.594046 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.594023    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 09 20:43:32.594276 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.594256    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:43:32.594357 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.594275    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:43:32.601318 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.601297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:43:32.649194 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.649154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.649347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.649206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.649347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.649235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdzd\" (UniqueName: \"kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.649347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.649266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.649347 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.649300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750332 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdzd\" (UniqueName: \"kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750534 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750674 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750745 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.750745 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.750719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.751282 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.751254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.751390 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.751264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.752976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.752955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.758304 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.758279    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdzd\" (UniqueName: \"kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.864700 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.864608    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/1.log"
Sep 09 20:43:32.865080 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865060    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n_fa244737-6f97-4e2d-8c49-0e4d1c721b03/storage-initializer/0.log"
Sep 09 20:43:32.865135 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865107    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" containerID="7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab" exitCode=1
Sep 09 20:43:32.865183 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"
Sep 09 20:43:32.865217 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerDied","Data":"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"}
Sep 09 20:43:32.865264 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n" event={"ID":"fa244737-6f97-4e2d-8c49-0e4d1c721b03","Type":"ContainerDied","Data":"1e9e7b3f40590ceecd189ea8d578c7cfd7eb85e63f1db91672d0a88be0ae3585"}
Sep 09 20:43:32.865300 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.865292    2582 scope.go:117] "RemoveContainer" containerID="7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"
Sep 09 20:43:32.873672 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.873653    2582 scope.go:117] "RemoveContainer" containerID="0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"
Sep 09 20:43:32.881014 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.880996    2582 scope.go:117] "RemoveContainer" containerID="7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"
Sep 09 20:43:32.881262 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:32.881246    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab\": container with ID starting with 7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab not found: ID does not exist" containerID="7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"
Sep 09 20:43:32.881306 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.881268    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab"} err="failed to get container status \"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab\": rpc error: code = NotFound desc = could not find container \"7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab\": container with ID starting with 7af30c21ecb20392964f82d2f56f428dad523501b5f57c54c4fcdadc264f31ab not found: ID does not exist"
Sep 09 20:43:32.881306 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.881285    2582 scope.go:117] "RemoveContainer" containerID="0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"
Sep 09 20:43:32.881512 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:43:32.881493    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790\": container with ID starting with 0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790 not found: ID does not exist" containerID="0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"
Sep 09 20:43:32.881567 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.881521    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790"} err="failed to get container status \"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790\": rpc error: code = NotFound desc = could not find container \"0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790\": container with ID starting with 0a56e26963d74387f4985bccd994b81d6e0607c420e65443c1a8b7fbddf46790 not found: ID does not exist"
Sep 09 20:43:32.899924 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.899891    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:32.903337 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.903319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:32.906198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:32.906177    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-2jk7n"]
Sep 09 20:43:33.124802 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:33.124727    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa244737-6f97-4e2d-8c49-0e4d1c721b03" path="/var/lib/kubelet/pods/fa244737-6f97-4e2d-8c49-0e4d1c721b03/volumes"
Sep 09 20:43:33.175456 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:33.175427    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:43:33.178726 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:43:33.178648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a012b4_7eda_45eb_b5cd_197a6a14bd77.slice/crio-e636702c32e6dffea9b44b95582573d1ded0e6d80b2a87fd5040f9b84962fcef WatchSource:0}: Error finding container e636702c32e6dffea9b44b95582573d1ded0e6d80b2a87fd5040f9b84962fcef: Status 404 returned error can't find the container with id e636702c32e6dffea9b44b95582573d1ded0e6d80b2a87fd5040f9b84962fcef
Sep 09 20:43:33.873610 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:33.873570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerStarted","Data":"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"}
Sep 09 20:43:33.874039 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:33.873628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerStarted","Data":"e636702c32e6dffea9b44b95582573d1ded0e6d80b2a87fd5040f9b84962fcef"}
Sep 09 20:43:34.885914 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:34.885874    2582 generic.go:356] "Generic (PLEG): container finished" podID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerID="b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5" exitCode=0
Sep 09 20:43:34.886301 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:34.885957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerDied","Data":"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"}
Sep 09 20:43:35.893172 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:35.893140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerStarted","Data":"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"}
Sep 09 20:43:35.893560 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:35.893192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerStarted","Data":"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"}
Sep 09 20:43:35.893560 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:35.893387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:35.914451 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:35.914401    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podStartSLOduration=2.9085766189999998 podStartE2EDuration="3.914386371s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.005809752s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:35.912018258 +0000 UTC m=+2905.312695450" watchObservedRunningTime="2026-09-09 20:43:35.914386371 +0000 UTC m=+2905.315063637"
Sep 09 20:43:36.898431 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:36.898392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:36.899869 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:36.899838    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:43:37.902212 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:37.902171    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:43:42.906944 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:42.906915    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:43:42.907453 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:42.907423    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:43:52.907468 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:43:52.907422    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:02.907657 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:02.907614    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:12.907603 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:12.907557    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:22.907853 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:22.907804    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:32.907648 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:32.907608    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:42.907496 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:42.907456    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:44:52.907868 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:44:52.907834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:45:02.644171 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:02.644129    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:45:02.644795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:02.644425    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" containerID="cri-o://8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0" gracePeriod=30
Sep 09 20:45:02.644795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:02.644472    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kube-rbac-proxy" containerID="cri-o://8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358" gracePeriod=30
Sep 09 20:45:02.902589 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:02.902490    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 09 20:45:02.907858 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:02.907835    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 09 20:45:03.284411 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.284377    2582 generic.go:356] "Generic (PLEG): container finished" podID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerID="8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358" exitCode=2
Sep 09 20:45:03.284569 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.284453    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerDied","Data":"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"}
Sep 09 20:45:03.766438 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.766404    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:03.770029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.770009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:03.774903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.774873    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 09 20:45:03.774903 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.774900    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 09 20:45:03.781513 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.781491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:03.910955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.910913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:03.910955 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.910954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:03.911205 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.910995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:03.911205 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:03.911094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnh9\" (UniqueName: \"kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.012475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.012441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.012475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.012475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.012675 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.012498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.012675 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.012554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnh9\" (UniqueName: \"kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.012899 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.012880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.013230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.013209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.014941 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.014918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.020353 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.020295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnh9\" (UniqueName: \"kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.081251 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.081225    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:04.355886 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:04.355858    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:04.358142 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:45:04.358111    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode517612b_40e2_4336_89cb_326d59dbe782.slice/crio-dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9 WatchSource:0}: Error finding container dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9: Status 404 returned error can't find the container with id dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9
Sep 09 20:45:05.294797 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:05.294763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerStarted","Data":"ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"}
Sep 09 20:45:05.295170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:05.294824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerStarted","Data":"dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9"}
Sep 09 20:45:07.291324 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.291302    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:45:07.306609 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.306555    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/0.log"
Sep 09 20:45:07.306609 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.306589    2582 generic.go:356] "Generic (PLEG): container finished" podID="e517612b-40e2-4336-89cb-326d59dbe782" containerID="ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d" exitCode=1
Sep 09 20:45:07.306786 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.306664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerDied","Data":"ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"}
Sep 09 20:45:07.308719 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.308697    2582 generic.go:356] "Generic (PLEG): container finished" podID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerID="8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0" exitCode=0
Sep 09 20:45:07.308823 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.308761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerDied","Data":"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"}
Sep 09 20:45:07.308823 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.308787    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"
Sep 09 20:45:07.308823 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.308816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz" event={"ID":"22a012b4-7eda-45eb-b5cd-197a6a14bd77","Type":"ContainerDied","Data":"e636702c32e6dffea9b44b95582573d1ded0e6d80b2a87fd5040f9b84962fcef"}
Sep 09 20:45:07.308984 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.308859    2582 scope.go:117] "RemoveContainer" containerID="8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"
Sep 09 20:45:07.319535 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.319517    2582 scope.go:117] "RemoveContainer" containerID="8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"
Sep 09 20:45:07.333855 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.333839    2582 scope.go:117] "RemoveContainer" containerID="b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"
Sep 09 20:45:07.346253 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.346232    2582 scope.go:117] "RemoveContainer" containerID="8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"
Sep 09 20:45:07.346496 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:45:07.346477    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358\": container with ID starting with 8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358 not found: ID does not exist" containerID="8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"
Sep 09 20:45:07.346556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.346503    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358"} err="failed to get container status \"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358\": rpc error: code = NotFound desc = could not find container \"8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358\": container with ID starting with 8a5f2686a9fbc4569249ec0ac6d9a066ab5f0cebac545ef431bb89a2d7061358 not found: ID does not exist"
Sep 09 20:45:07.346556 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.346522    2582 scope.go:117] "RemoveContainer" containerID="8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"
Sep 09 20:45:07.346801 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:45:07.346784    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0\": container with ID starting with 8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0 not found: ID does not exist" containerID="8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"
Sep 09 20:45:07.346898 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.346804    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0"} err="failed to get container status \"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0\": rpc error: code = NotFound desc = could not find container \"8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0\": container with ID starting with 8ee22be0935fe5b44a1d5c13ba1ccd35c2720a114740ff7f30507c56f00554d0 not found: ID does not exist"
Sep 09 20:45:07.346898 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.346818    2582 scope.go:117] "RemoveContainer" containerID="b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"
Sep 09 20:45:07.347133 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:45:07.347105    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5\": container with ID starting with b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5 not found: ID does not exist" containerID="b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"
Sep 09 20:45:07.347196 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.347137    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5"} err="failed to get container status \"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5\": rpc error: code = NotFound desc = could not find container \"b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5\": container with ID starting with b9ab62a325aca79fbf4f35a3edf96983c2e3ad40d2f29f6f4ac5227a89a3d7c5 not found: ID does not exist"
Sep 09 20:45:07.441206 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrdzd\" (UniqueName: \"kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd\") pod \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") "
Sep 09 20:45:07.441372 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location\") pod \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") "
Sep 09 20:45:07.441372 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls\") pod \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") "
Sep 09 20:45:07.441372 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert\") pod \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") "
Sep 09 20:45:07.441549 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441397    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config\") pod \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\" (UID: \"22a012b4-7eda-45eb-b5cd-197a6a14bd77\") "
Sep 09 20:45:07.441715 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22a012b4-7eda-45eb-b5cd-197a6a14bd77" (UID: "22a012b4-7eda-45eb-b5cd-197a6a14bd77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:07.441858 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441779    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:07.441858 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.441787    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22a012b4-7eda-45eb-b5cd-197a6a14bd77" (UID: "22a012b4-7eda-45eb-b5cd-197a6a14bd77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:07.442151 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.442117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22a012b4-7eda-45eb-b5cd-197a6a14bd77" (UID: "22a012b4-7eda-45eb-b5cd-197a6a14bd77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:07.443517 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.443493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22a012b4-7eda-45eb-b5cd-197a6a14bd77" (UID: "22a012b4-7eda-45eb-b5cd-197a6a14bd77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:07.443603 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.443528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd" (OuterVolumeSpecName: "kube-api-access-jrdzd") pod "22a012b4-7eda-45eb-b5cd-197a6a14bd77" (UID: "22a012b4-7eda-45eb-b5cd-197a6a14bd77"). InnerVolumeSpecName "kube-api-access-jrdzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:07.543238 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.543201    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a012b4-7eda-45eb-b5cd-197a6a14bd77-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:07.543238 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.543232    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:07.543443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.543248    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:07.543443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.543262    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdzd\" (UniqueName: \"kubernetes.io/projected/22a012b4-7eda-45eb-b5cd-197a6a14bd77-kube-api-access-jrdzd\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:07.631433 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.631399    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:45:07.636192 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:07.636167    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-2m7mz"]
Sep 09 20:45:08.314693 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:08.314607    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/0.log"
Sep 09 20:45:08.315076 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:08.314722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerStarted","Data":"5357dd4b14910d74a8c2ff1de78527720a51f6753023a43fbb37a0fff3eeec34"}
Sep 09 20:45:09.124743 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:09.124709    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" path="/var/lib/kubelet/pods/22a012b4-7eda-45eb-b5cd-197a6a14bd77/volumes"
Sep 09 20:45:12.334957 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.334929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/1.log"
Sep 09 20:45:12.335395 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.335334    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/0.log"
Sep 09 20:45:12.335395 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.335377    2582 generic.go:356] "Generic (PLEG): container finished" podID="e517612b-40e2-4336-89cb-326d59dbe782" containerID="5357dd4b14910d74a8c2ff1de78527720a51f6753023a43fbb37a0fff3eeec34" exitCode=1
Sep 09 20:45:12.335491 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.335451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerDied","Data":"5357dd4b14910d74a8c2ff1de78527720a51f6753023a43fbb37a0fff3eeec34"}
Sep 09 20:45:12.335537 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.335525    2582 scope.go:117] "RemoveContainer" containerID="ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"
Sep 09 20:45:12.335860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.335844    2582 scope.go:117] "RemoveContainer" containerID="ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"
Sep 09 20:45:12.347456 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:45:12.347436    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_kserve-ci-e2e-test(e517612b-40e2-4336-89cb-326d59dbe782)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" podUID="e517612b-40e2-4336-89cb-326d59dbe782"
Sep 09 20:45:12.347597 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:45:12.347572    2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_kserve-ci-e2e-test_e517612b-40e2-4336-89cb-326d59dbe782_0 in pod sandbox dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9 from index: no such id: 'ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d'" containerID="ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"
Sep 09 20:45:12.347663 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:12.347610    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_kserve-ci-e2e-test_e517612b-40e2-4336-89cb-326d59dbe782_0 in pod sandbox dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9 from index: no such id: 'ce576e1f1ee55415aec4c9a76ca8d73cf16429fda92e9a6e66a1574fd23e896d'"
Sep 09 20:45:13.341461 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.341432    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/1.log"
Sep 09 20:45:13.695346 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.695264    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:13.823094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.823074    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/1.log"
Sep 09 20:45:13.826444 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.826427    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:13.892406 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location\") pod \"e517612b-40e2-4336-89cb-326d59dbe782\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") "
Sep 09 20:45:13.892544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892424    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls\") pod \"e517612b-40e2-4336-89cb-326d59dbe782\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") "
Sep 09 20:45:13.892544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfnh9\" (UniqueName: \"kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9\") pod \"e517612b-40e2-4336-89cb-326d59dbe782\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") "
Sep 09 20:45:13.892544 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config\") pod \"e517612b-40e2-4336-89cb-326d59dbe782\" (UID: \"e517612b-40e2-4336-89cb-326d59dbe782\") "
Sep 09 20:45:13.892665 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e517612b-40e2-4336-89cb-326d59dbe782" (UID: "e517612b-40e2-4336-89cb-326d59dbe782"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:13.892937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.892912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e517612b-40e2-4336-89cb-326d59dbe782" (UID: "e517612b-40e2-4336-89cb-326d59dbe782"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:13.894381 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.894362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e517612b-40e2-4336-89cb-326d59dbe782" (UID: "e517612b-40e2-4336-89cb-326d59dbe782"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:13.894487 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.894470    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9" (OuterVolumeSpecName: "kube-api-access-cfnh9") pod "e517612b-40e2-4336-89cb-326d59dbe782" (UID: "e517612b-40e2-4336-89cb-326d59dbe782"). InnerVolumeSpecName "kube-api-access-cfnh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:13.993537 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.993513    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e517612b-40e2-4336-89cb-326d59dbe782-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:13.993655 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.993541    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e517612b-40e2-4336-89cb-326d59dbe782-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:13.993655 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.993551    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e517612b-40e2-4336-89cb-326d59dbe782-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:13.993655 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:13.993560    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnh9\" (UniqueName: \"kubernetes.io/projected/e517612b-40e2-4336-89cb-326d59dbe782-kube-api-access-cfnh9\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:45:14.353208 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.353124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz_e517612b-40e2-4336-89cb-326d59dbe782/storage-initializer/1.log"
Sep 09 20:45:14.353664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.353224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz" event={"ID":"e517612b-40e2-4336-89cb-326d59dbe782","Type":"ContainerDied","Data":"dadcb7d2f13988be53018ca445ebd0464f5eebcaaa44e87eb66f28736ec078d9"}
Sep 09 20:45:14.353664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.353244    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"
Sep 09 20:45:14.353664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.353310    2582 scope.go:117] "RemoveContainer" containerID="5357dd4b14910d74a8c2ff1de78527720a51f6753023a43fbb37a0fff3eeec34"
Sep 09 20:45:14.395178 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.395148    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:14.398107 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.398082    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-n8bdz"]
Sep 09 20:45:14.862983 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.862954    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:45:14.863346 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863331    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="storage-initializer"
Sep 09 20:45:14.863389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863348    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="storage-initializer"
Sep 09 20:45:14.863389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863364    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container"
Sep 09 20:45:14.863389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863372    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container"
Sep 09 20:45:14.863389 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863387    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863394    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863400    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863406    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863412    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kube-rbac-proxy"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kube-rbac-proxy"
Sep 09 20:45:14.863507 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863493    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kube-rbac-proxy"
Sep 09 20:45:14.863710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863509    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a012b4-7eda-45eb-b5cd-197a6a14bd77" containerName="kserve-container"
Sep 09 20:45:14.863710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863518    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.863710 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.863658    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e517612b-40e2-4336-89cb-326d59dbe782" containerName="storage-initializer"
Sep 09 20:45:14.868289 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.868267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:14.871189 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:45:14.871492 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871471    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:45:14.871570 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871511    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:45:14.871790 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871776    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:45:14.871960 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 09 20:45:14.872013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.871985    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:45:14.872403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.872388    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:45:14.877400 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.877376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:45:14.899730 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.899703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:14.899828 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.899757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:14.899828 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.899779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4tf\" (UniqueName: \"kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:14.899918 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.899829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:14.899918 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:14.899886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001062 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001247 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001247 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001247 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001247 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4tf\" (UniqueName: \"kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.001937 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.002013 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.001950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.003614 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.003594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.010385 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.010366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4tf\" (UniqueName: \"kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf\") pod \"isvc-pmml-runtime-predictor-885fc8777-jbzl6\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.125424 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.125333    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e517612b-40e2-4336-89cb-326d59dbe782" path="/var/lib/kubelet/pods/e517612b-40e2-4336-89cb-326d59dbe782/volumes"
Sep 09 20:45:15.180059 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.180032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:15.452894 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:15.452871    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:45:15.455488 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:45:15.455453    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb873e3_dd12_431b_b709_7c6101edbccd.slice/crio-a93e12548ae8d585d1c9c5aac21bc127bb158f2a49471968ea9e6099b9c16739 WatchSource:0}: Error finding container a93e12548ae8d585d1c9c5aac21bc127bb158f2a49471968ea9e6099b9c16739: Status 404 returned error can't find the container with id a93e12548ae8d585d1c9c5aac21bc127bb158f2a49471968ea9e6099b9c16739
Sep 09 20:45:16.366889 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:16.366854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerStarted","Data":"f6c358539f50ca60822e5e9edc3b6429d1380de7d0bd562defbe82f8000f71c0"}
Sep 09 20:45:16.367062 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:16.366913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerStarted","Data":"a93e12548ae8d585d1c9c5aac21bc127bb158f2a49471968ea9e6099b9c16739"}
Sep 09 20:45:19.382339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:19.382256    2582 generic.go:356] "Generic (PLEG): container finished" podID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerID="f6c358539f50ca60822e5e9edc3b6429d1380de7d0bd562defbe82f8000f71c0" exitCode=0
Sep 09 20:45:19.382339 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:19.382327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerDied","Data":"f6c358539f50ca60822e5e9edc3b6429d1380de7d0bd562defbe82f8000f71c0"}
Sep 09 20:45:26.425213 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.425184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerStarted","Data":"9f7f2b904cc222d5f0233ab646ca7dbb4aae6672ef43f5150991b9aa44d8c373"}
Sep 09 20:45:26.425559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.425277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerStarted","Data":"44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f"}
Sep 09 20:45:26.425628 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.425588    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:26.425721 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.425709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:26.427179 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.427153    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:45:26.447132 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:26.447080    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podStartSLOduration=1.9678252889999999 podStartE2EDuration="12.447065963s" totalImagesPullingTime="6.895211766s" totalInitContainerRuntime="3.584028908s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:26.444083065 +0000 UTC m=+3015.844760257" watchObservedRunningTime="2026-09-09 20:45:26.447065963 +0000 UTC m=+3015.847743151"
Sep 09 20:45:27.430326 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:27.430289    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:45:32.434529 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:32.434496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:45:32.434979 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:32.434932    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:45:42.435015 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:42.434974    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:45:52.434962 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:45:52.434922    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:02.435137 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:02.435077    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:12.435191 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:12.435089    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:22.435379 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:22.435337    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:32.434968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:32.434923    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:42.434911 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:42.434871    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:49.125071 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:49.124995    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:46:56.157578 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:56.157537    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:46:56.158153 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:56.157777    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" containerID="cri-o://44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f" gracePeriod=30
Sep 09 20:46:56.158153 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:56.157836    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kube-rbac-proxy" containerID="cri-o://9f7f2b904cc222d5f0233ab646ca7dbb4aae6672ef43f5150991b9aa44d8c373" gracePeriod=30
Sep 09 20:46:56.836021 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:56.835985    2582 generic.go:356] "Generic (PLEG): container finished" podID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerID="9f7f2b904cc222d5f0233ab646ca7dbb4aae6672ef43f5150991b9aa44d8c373" exitCode=2
Sep 09 20:46:56.836190 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:56.836051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerDied","Data":"9f7f2b904cc222d5f0233ab646ca7dbb4aae6672ef43f5150991b9aa44d8c373"}
Sep 09 20:46:57.430910 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:57.430868    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 09 20:46:59.119798 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.119756    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 09 20:46:59.506086 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:46:59.506055    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb873e3_dd12_431b_b709_7c6101edbccd.slice/crio-44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb873e3_dd12_431b_b709_7c6101edbccd.slice/crio-conmon-44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 20:46:59.854501 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.854469    2582 generic.go:356] "Generic (PLEG): container finished" podID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerID="44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f" exitCode=0
Sep 09 20:46:59.854649 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.854552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerDied","Data":"44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f"}
Sep 09 20:46:59.896673 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.896647    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:46:59.993797 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.993756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert\") pod \"7cb873e3-dd12-431b-b709-7c6101edbccd\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") "
Sep 09 20:46:59.993974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.993821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config\") pod \"7cb873e3-dd12-431b-b709-7c6101edbccd\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") "
Sep 09 20:46:59.993974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.993865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls\") pod \"7cb873e3-dd12-431b-b709-7c6101edbccd\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") "
Sep 09 20:46:59.993974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.993896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location\") pod \"7cb873e3-dd12-431b-b709-7c6101edbccd\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") "
Sep 09 20:46:59.993974 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.993925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc4tf\" (UniqueName: \"kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf\") pod \"7cb873e3-dd12-431b-b709-7c6101edbccd\" (UID: \"7cb873e3-dd12-431b-b709-7c6101edbccd\") "
Sep 09 20:46:59.994191 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.994160    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cb873e3-dd12-431b-b709-7c6101edbccd" (UID: "7cb873e3-dd12-431b-b709-7c6101edbccd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:59.994254 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.994215    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cb873e3-dd12-431b-b709-7c6101edbccd" (UID: "7cb873e3-dd12-431b-b709-7c6101edbccd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:59.994254 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.994237    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cb873e3-dd12-431b-b709-7c6101edbccd" (UID: "7cb873e3-dd12-431b-b709-7c6101edbccd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:59.996017 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.995992    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cb873e3-dd12-431b-b709-7c6101edbccd" (UID: "7cb873e3-dd12-431b-b709-7c6101edbccd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:59.996092 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:46:59.995997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf" (OuterVolumeSpecName: "kube-api-access-fc4tf") pod "7cb873e3-dd12-431b-b709-7c6101edbccd" (UID: "7cb873e3-dd12-431b-b709-7c6101edbccd"). InnerVolumeSpecName "kube-api-access-fc4tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:00.095245 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.095145    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:47:00.095245 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.095179    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:47:00.095245 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.095193    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cb873e3-dd12-431b-b709-7c6101edbccd-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:47:00.095245 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.095206    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb873e3-dd12-431b-b709-7c6101edbccd-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:47:00.095245 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.095220    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc4tf\" (UniqueName: \"kubernetes.io/projected/7cb873e3-dd12-431b-b709-7c6101edbccd-kube-api-access-fc4tf\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:47:00.862579 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.862539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6" event={"ID":"7cb873e3-dd12-431b-b709-7c6101edbccd","Type":"ContainerDied","Data":"a93e12548ae8d585d1c9c5aac21bc127bb158f2a49471968ea9e6099b9c16739"}
Sep 09 20:47:00.863016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.862616    2582 scope.go:117] "RemoveContainer" containerID="9f7f2b904cc222d5f0233ab646ca7dbb4aae6672ef43f5150991b9aa44d8c373"
Sep 09 20:47:00.863016 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.862550    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"
Sep 09 20:47:00.871827 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.871802    2582 scope.go:117] "RemoveContainer" containerID="44a8096048d9b0c4428fb8c0c147b3833444edbfb7748e857ae41925d1e7295f"
Sep 09 20:47:00.878982 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.878965    2582 scope.go:117] "RemoveContainer" containerID="f6c358539f50ca60822e5e9edc3b6429d1380de7d0bd562defbe82f8000f71c0"
Sep 09 20:47:00.889241 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.889221    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:47:00.897907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:00.897886    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jbzl6"]
Sep 09 20:47:01.124470 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.124390    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" path="/var/lib/kubelet/pods/7cb873e3-dd12-431b-b709-7c6101edbccd/volumes"
Sep 09 20:47:01.486158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486124    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:47:01.486508 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486496    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="storage-initializer"
Sep 09 20:47:01.486559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486510    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="storage-initializer"
Sep 09 20:47:01.486559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486520    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kube-rbac-proxy"
Sep 09 20:47:01.486559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486525    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kube-rbac-proxy"
Sep 09 20:47:01.486559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486546    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container"
Sep 09 20:47:01.486559 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486551    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container"
Sep 09 20:47:01.486765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486605    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kube-rbac-proxy"
Sep 09 20:47:01.486765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.486614    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb873e3-dd12-431b-b709-7c6101edbccd" containerName="kserve-container"
Sep 09 20:47:01.491250 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.491219    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.495618 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.495597    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 09 20:47:01.495861 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.495845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:47:01.501833 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.501813    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:47:01.501930 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.501831    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:47:01.501995 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.501946    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 20:47:01.501995 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.501968    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:47:01.502099 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.502042    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:47:01.507021 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.506998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.507113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.507070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljcb\" (UniqueName: \"kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.507113 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.507101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.507221 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.507203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.507314 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.507256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.607987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.607953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljcb\" (UniqueName: \"kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.607996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608136 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608318 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608318 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:47:01.608169    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 09 20:47:01.608318 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:47:01.608245    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls podName:4ac8d0f9-ebb5-4418-93b6-6fa29079a922 nodeName:}" failed. No retries permitted until 2026-09-09 20:47:02.108223313 +0000 UTC m=+3111.508900498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 09 20:47:01.608475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608457    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608762 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608746    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.608805 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.608770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:01.627691 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.627661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:47:01.636488 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:01.636459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljcb\" (UniqueName: \"kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:02.111754 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.111712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:02.114235 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.114210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:02.401577 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.401487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:02.692256 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.692094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:47:02.696949 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:47:02.696921    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac8d0f9_ebb5_4418_93b6_6fa29079a922.slice/crio-d8c614aa73446aedea8d58bc20a840e092434a88e60f405557d1ef4836aabb01 WatchSource:0}: Error finding container d8c614aa73446aedea8d58bc20a840e092434a88e60f405557d1ef4836aabb01: Status 404 returned error can't find the container with id d8c614aa73446aedea8d58bc20a840e092434a88e60f405557d1ef4836aabb01
Sep 09 20:47:02.875029 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.874991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerStarted","Data":"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"}
Sep 09 20:47:02.875294 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:02.875060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerStarted","Data":"d8c614aa73446aedea8d58bc20a840e092434a88e60f405557d1ef4836aabb01"}
Sep 09 20:47:06.894802 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:06.894710    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerID="fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d" exitCode=0
Sep 09 20:47:06.894802 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:06.894785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerDied","Data":"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"}
Sep 09 20:47:07.901431 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:07.901399    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerStarted","Data":"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"}
Sep 09 20:47:07.901825 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:07.901458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerStarted","Data":"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"}
Sep 09 20:47:07.901825 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:07.901650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:08.906158 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:08.906121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:08.907481 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:08.907452    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:09.910873 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:09.910829    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:14.916054 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:14.916023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:47:14.916582 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:14.916552    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:14.964167 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:14.964116    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podStartSLOduration=10.099171442 podStartE2EDuration="13.964102071s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864930629s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:07.964942603 +0000 UTC m=+3117.365619794" watchObservedRunningTime="2026-09-09 20:47:14.964102071 +0000 UTC m=+3124.364779266"
Sep 09 20:47:24.916722 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:24.916665    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:34.916785 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:34.916703    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:44.916479 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:44.916438    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:47:54.917467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:47:54.917418    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:48:04.916765 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:04.916723    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:48:14.916727 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:14.916666    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:48:24.917328 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:24.917295    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:48:32.371697 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:32.371649    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:48:32.372062 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:32.371913    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" containerID="cri-o://5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370" gracePeriod=30
Sep 09 20:48:32.372062 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:32.371982    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kube-rbac-proxy" containerID="cri-o://ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df" gracePeriod=30
Sep 09 20:48:33.293647 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:33.293610    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerID="ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df" exitCode=2
Sep 09 20:48:33.293825 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:33.293663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerDied","Data":"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"}
Sep 09 20:48:34.911881 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:34.911840    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 09 20:48:34.916443 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:34.916418    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 09 20:48:36.118315 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.118288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:48:36.203742 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.203714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lljcb\" (UniqueName: \"kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb\") pod \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") "
Sep 09 20:48:36.203931 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.203779    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location\") pod \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") "
Sep 09 20:48:36.203931 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.203796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config\") pod \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") "
Sep 09 20:48:36.203931 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.203829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert\") pod \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") "
Sep 09 20:48:36.203931 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.203856    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") pod \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\" (UID: \"4ac8d0f9-ebb5-4418-93b6-6fa29079a922\") "
Sep 09 20:48:36.204195 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.204165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ac8d0f9-ebb5-4418-93b6-6fa29079a922" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:36.204252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.204172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ac8d0f9-ebb5-4418-93b6-6fa29079a922" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:36.204252 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.204199    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ac8d0f9-ebb5-4418-93b6-6fa29079a922" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:36.205901 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.205877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb" (OuterVolumeSpecName: "kube-api-access-lljcb") pod "4ac8d0f9-ebb5-4418-93b6-6fa29079a922" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922"). InnerVolumeSpecName "kube-api-access-lljcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:36.205968 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.205933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ac8d0f9-ebb5-4418-93b6-6fa29079a922" (UID: "4ac8d0f9-ebb5-4418-93b6-6fa29079a922"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:48:36.304860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.304772    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:48:36.304860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.304799    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:48:36.304860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.304813    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:48:36.304860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.304825    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:48:36.304860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.304838    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljcb\" (UniqueName: \"kubernetes.io/projected/4ac8d0f9-ebb5-4418-93b6-6fa29079a922-kube-api-access-lljcb\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:48:36.308796 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.308768    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerID="5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370" exitCode=0
Sep 09 20:48:36.308896 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.308823    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"
Sep 09 20:48:36.308896 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.308834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerDied","Data":"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"}
Sep 09 20:48:36.308896 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.308881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q" event={"ID":"4ac8d0f9-ebb5-4418-93b6-6fa29079a922","Type":"ContainerDied","Data":"d8c614aa73446aedea8d58bc20a840e092434a88e60f405557d1ef4836aabb01"}
Sep 09 20:48:36.309003 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.308906    2582 scope.go:117] "RemoveContainer" containerID="ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"
Sep 09 20:48:36.318321 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.318305    2582 scope.go:117] "RemoveContainer" containerID="5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"
Sep 09 20:48:36.325668 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.325650    2582 scope.go:117] "RemoveContainer" containerID="fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"
Sep 09 20:48:36.332759 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.332740    2582 scope.go:117] "RemoveContainer" containerID="ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"
Sep 09 20:48:36.332999 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:48:36.332981    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df\": container with ID starting with ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df not found: ID does not exist" containerID="ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"
Sep 09 20:48:36.333050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.333007    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df"} err="failed to get container status \"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df\": rpc error: code = NotFound desc = could not find container \"ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df\": container with ID starting with ca0f2458d2e235aa974f97b240abc47521c6cf13a6a3e60b69e4fe674e6584df not found: ID does not exist"
Sep 09 20:48:36.333050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.333024    2582 scope.go:117] "RemoveContainer" containerID="5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"
Sep 09 20:48:36.333272 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:48:36.333253    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370\": container with ID starting with 5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370 not found: ID does not exist" containerID="5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"
Sep 09 20:48:36.333317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.333281    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370"} err="failed to get container status \"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370\": rpc error: code = NotFound desc = could not find container \"5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370\": container with ID starting with 5af3857cfd6bbb904478151f5afbcc81eb96238fcd260eec096499a3444fe370 not found: ID does not exist"
Sep 09 20:48:36.333317 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.333298    2582 scope.go:117] "RemoveContainer" containerID="fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"
Sep 09 20:48:36.333542 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:48:36.333528    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d\": container with ID starting with fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d not found: ID does not exist" containerID="fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"
Sep 09 20:48:36.333592 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.333543    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d"} err="failed to get container status \"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d\": rpc error: code = NotFound desc = could not find container \"fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d\": container with ID starting with fce40719c48294a563c86b3207f63569c12bad32872eec0d53b41224eda2230d not found: ID does not exist"
Sep 09 20:48:36.340753 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.340730    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:48:36.351487 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:36.351465    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pff6q"]
Sep 09 20:48:37.124505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.124471    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" path="/var/lib/kubelet/pods/4ac8d0f9-ebb5-4418-93b6-6fa29079a922/volumes"
Sep 09 20:48:37.542495 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542458    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:48:37.542843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542821    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="storage-initializer"
Sep 09 20:48:37.542843 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542835    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="storage-initializer"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542851    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542858    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542865    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kube-rbac-proxy"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542870    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kube-rbac-proxy"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542927    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kube-rbac-proxy"
Sep 09 20:48:37.543002 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.542937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac8d0f9-ebb5-4418-93b6-6fa29079a922" containerName="kserve-container"
Sep 09 20:48:37.547565 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.547544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.550298 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.550277    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:48:37.551239 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551191    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:48:37.551239 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:48:37.551427 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:48:37.551427 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551291    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:48:37.551427 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-887399-predictor-serving-cert\""
Sep 09 20:48:37.551766 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.551751    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-887399-kube-rbac-proxy-sar-config\""
Sep 09 20:48:37.559757 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.559735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:48:37.716148 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.716107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.716342 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.716168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.716342 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.716212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.716342 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.716236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.716478 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.716338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblg9\" (UniqueName: \"kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817376 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblg9\" (UniqueName: \"kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817376 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817613 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817613 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817613 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.817902 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.817876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.818087 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.818063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.818170 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.818153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.819957 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.819932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.830123 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.830104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblg9\" (UniqueName: \"kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9\") pod \"isvc-primary-887399-predictor-5fd8bb46b6-jddn2\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") " pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:37.857651 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:37.857627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:38.135661 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:38.135636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:48:38.137642 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:48:38.137616    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be4340d_5b11_4fa2_808b_3f62629006c6.slice/crio-1b7cf6e04a52f3fb6e3796ae30ef254342f266f20b7d98decb892d5cbfdf8768 WatchSource:0}: Error finding container 1b7cf6e04a52f3fb6e3796ae30ef254342f266f20b7d98decb892d5cbfdf8768: Status 404 returned error can't find the container with id 1b7cf6e04a52f3fb6e3796ae30ef254342f266f20b7d98decb892d5cbfdf8768
Sep 09 20:48:38.321004 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:38.320969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerStarted","Data":"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"}
Sep 09 20:48:38.321168 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:38.321035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerStarted","Data":"1b7cf6e04a52f3fb6e3796ae30ef254342f266f20b7d98decb892d5cbfdf8768"}
Sep 09 20:48:42.339664 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:42.339624    2582 generic.go:356] "Generic (PLEG): container finished" podID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerID="57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18" exitCode=0
Sep 09 20:48:42.340050 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:42.339708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerDied","Data":"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"}
Sep 09 20:48:43.346279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:43.346246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerStarted","Data":"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"}
Sep 09 20:48:43.346709 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:43.346334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerStarted","Data":"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"}
Sep 09 20:48:43.346709 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:43.346561    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:43.346709 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:43.346619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:43.347913 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:43.347873    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:48:44.351475 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:44.351432    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:48:49.355897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:49.355862    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:48:49.356351 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:49.356325    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:48:49.392497 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:49.392446    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podStartSLOduration=8.360305323 podStartE2EDuration="12.392431494s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.032126171s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:43.385598146 +0000 UTC m=+3212.786275338" watchObservedRunningTime="2026-09-09 20:48:49.392431494 +0000 UTC m=+3218.793108685"
Sep 09 20:48:59.357336 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:48:59.357293    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:09.356480 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:09.356397    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:19.356907 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:19.356864    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:29.357224 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:29.357177    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:39.356351 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:39.356311    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:49.356963 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:49.356924    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 09 20:49:59.356849 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:49:59.356817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:50:42.781857 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:42.781821    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:50:42.782352 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:42.782061    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container" containerID="cri-o://0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a" gracePeriod=30
Sep 09 20:50:42.782352 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:42.782089    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kube-rbac-proxy" containerID="cri-o://59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff" gracePeriod=30
Sep 09 20:50:43.887230 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:43.887189    2582 generic.go:356] "Generic (PLEG): container finished" podID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerID="59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff" exitCode=2
Sep 09 20:50:43.887620 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:43.887238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerDied","Data":"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"}
Sep 09 20:50:44.351795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:44.351749    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Sep 09 20:50:47.425006 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.424975    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:50:47.531279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls\") pod \"8be4340d-5b11-4fa2-808b-3f62629006c6\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") "
Sep 09 20:50:47.531279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531235    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rblg9\" (UniqueName: \"kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9\") pod \"8be4340d-5b11-4fa2-808b-3f62629006c6\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") "
Sep 09 20:50:47.531279 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location\") pod \"8be4340d-5b11-4fa2-808b-3f62629006c6\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") "
Sep 09 20:50:47.531577 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531291    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config\") pod \"8be4340d-5b11-4fa2-808b-3f62629006c6\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") "
Sep 09 20:50:47.531577 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531342    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert\") pod \"8be4340d-5b11-4fa2-808b-3f62629006c6\" (UID: \"8be4340d-5b11-4fa2-808b-3f62629006c6\") "
Sep 09 20:50:47.531577 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531547    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8be4340d-5b11-4fa2-808b-3f62629006c6" (UID: "8be4340d-5b11-4fa2-808b-3f62629006c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:47.531711 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531646    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8be4340d-5b11-4fa2-808b-3f62629006c6" (UID: "8be4340d-5b11-4fa2-808b-3f62629006c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:47.531756 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.531710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8be4340d-5b11-4fa2-808b-3f62629006c6" (UID: "8be4340d-5b11-4fa2-808b-3f62629006c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:47.533294 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.533272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8be4340d-5b11-4fa2-808b-3f62629006c6" (UID: "8be4340d-5b11-4fa2-808b-3f62629006c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:47.533392 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.533336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9" (OuterVolumeSpecName: "kube-api-access-rblg9") pod "8be4340d-5b11-4fa2-808b-3f62629006c6" (UID: "8be4340d-5b11-4fa2-808b-3f62629006c6"). InnerVolumeSpecName "kube-api-access-rblg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:47.632094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.632067    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:50:47.632094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.632092    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be4340d-5b11-4fa2-808b-3f62629006c6-cabundle-cert\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:50:47.632094 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.632101    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8be4340d-5b11-4fa2-808b-3f62629006c6-proxy-tls\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:50:47.632311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.632109    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblg9\" (UniqueName: \"kubernetes.io/projected/8be4340d-5b11-4fa2-808b-3f62629006c6-kube-api-access-rblg9\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:50:47.632311 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.632117    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be4340d-5b11-4fa2-808b-3f62629006c6-kserve-provision-location\") on node \"ip-10-0-135-16.ec2.internal\" DevicePath \"\""
Sep 09 20:50:47.907051 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.906960    2582 generic.go:356] "Generic (PLEG): container finished" podID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerID="0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a" exitCode=0
Sep 09 20:50:47.907051 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.907017    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"
Sep 09 20:50:47.907237 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.907051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerDied","Data":"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"}
Sep 09 20:50:47.907237 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.907118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2" event={"ID":"8be4340d-5b11-4fa2-808b-3f62629006c6","Type":"ContainerDied","Data":"1b7cf6e04a52f3fb6e3796ae30ef254342f266f20b7d98decb892d5cbfdf8768"}
Sep 09 20:50:47.907237 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.907146    2582 scope.go:117] "RemoveContainer" containerID="59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"
Sep 09 20:50:47.916940 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.916921    2582 scope.go:117] "RemoveContainer" containerID="0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"
Sep 09 20:50:47.924718 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.924698    2582 scope.go:117] "RemoveContainer" containerID="57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"
Sep 09 20:50:47.930561 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.930538    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:50:47.932829 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.932714    2582 scope.go:117] "RemoveContainer" containerID="59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"
Sep 09 20:50:47.933208 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:50:47.933185    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff\": container with ID starting with 59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff not found: ID does not exist" containerID="59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"
Sep 09 20:50:47.933310 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.933220    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff"} err="failed to get container status \"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff\": rpc error: code = NotFound desc = could not find container \"59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff\": container with ID starting with 59209dbd822473763d4124beaf1f1e21bd7f4dd6e138599ff1445d3b67b500ff not found: ID does not exist"
Sep 09 20:50:47.933310 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.933241    2582 scope.go:117] "RemoveContainer" containerID="0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"
Sep 09 20:50:47.933548 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:50:47.933524    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a\": container with ID starting with 0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a not found: ID does not exist" containerID="0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"
Sep 09 20:50:47.933598 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.933559    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a"} err="failed to get container status \"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a\": rpc error: code = NotFound desc = could not find container \"0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a\": container with ID starting with 0eaa1dbf7c671de82b6885f0d3ed0bec9d921b96e69c53c95f03355126637f1a not found: ID does not exist"
Sep 09 20:50:47.933598 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.933581    2582 scope.go:117] "RemoveContainer" containerID="57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"
Sep 09 20:50:47.933881 ip-10-0-135-16 kubenswrapper[2582]: E0909 20:50:47.933864    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18\": container with ID starting with 57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18 not found: ID does not exist" containerID="57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"
Sep 09 20:50:47.933947 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.933892    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18"} err="failed to get container status \"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18\": rpc error: code = NotFound desc = could not find container \"57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18\": container with ID starting with 57d25a4821afbb7863af6c46172e42b02284f13eb552ff5d9038070650e15a18 not found: ID does not exist"
Sep 09 20:50:47.934838 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:47.934818    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-887399-predictor-5fd8bb46b6-jddn2"]
Sep 09 20:50:49.124959 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:50:49.124925    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" path="/var/lib/kubelet/pods/8be4340d-5b11-4fa2-808b-3f62629006c6/volumes"
Sep 09 20:51:28.894797 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:28.894766    2582 ???:1] "http: TLS handshake error from 10.0.135.16:53060: EOF"
Sep 09 20:51:28.904350 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:28.904324    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wwrln_7c67ceb3-6e5e-4adf-8f2a-a443e346b239/global-pull-secret-syncer/0.log"
Sep 09 20:51:29.000751 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:29.000719    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-csjvb_25378334-29ae-4662-a4ba-9f4b126c8801/konnectivity-agent/0.log"
Sep 09 20:51:29.142508 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:29.142475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-16.ec2.internal_42b537ed4469da6dbc5578dc2b5d442c/haproxy/0.log"
Sep 09 20:51:32.926049 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:32.926015    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/alertmanager/0.log"
Sep 09 20:51:32.955416 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:32.955388    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/config-reloader/0.log"
Sep 09 20:51:32.990923 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:32.990898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/kube-rbac-proxy-web/0.log"
Sep 09 20:51:33.015761 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.015733    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/kube-rbac-proxy/0.log"
Sep 09 20:51:33.046383 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.046351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/kube-rbac-proxy-metric/0.log"
Sep 09 20:51:33.077512 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.077487    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/prom-label-proxy/0.log"
Sep 09 20:51:33.133018 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.132990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac1d7ef2-c581-4ee3-816b-8e14e50d9113/init-config-reloader/0.log"
Sep 09 20:51:33.631163 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.631127    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zrmwd_cef081dd-ded1-4f7c-94b9-7993b1566b2c/node-exporter/0.log"
Sep 09 20:51:33.659634 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.659605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zrmwd_cef081dd-ded1-4f7c-94b9-7993b1566b2c/kube-rbac-proxy/0.log"
Sep 09 20:51:33.684897 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.684867    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zrmwd_cef081dd-ded1-4f7c-94b9-7993b1566b2c/init-textfile/0.log"
Sep 09 20:51:33.717363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.717336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s92ft_9d7e48b6-9b20-425c-9821-9d11e3649a7b/kube-rbac-proxy-main/0.log"
Sep 09 20:51:33.743171 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.743144    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s92ft_9d7e48b6-9b20-425c-9821-9d11e3649a7b/kube-rbac-proxy-self/0.log"
Sep 09 20:51:33.768756 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:33.768731    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-s92ft_9d7e48b6-9b20-425c-9821-9d11e3649a7b/openshift-state-metrics/0.log"
Sep 09 20:51:34.174995 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.174961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d44f64c6-pwnwg_d6e63ebe-da2c-4833-aa6f-522980588a96/telemeter-client/0.log"
Sep 09 20:51:34.210777 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.210738    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d44f64c6-pwnwg_d6e63ebe-da2c-4833-aa6f-522980588a96/reload/0.log"
Sep 09 20:51:34.241198 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.241172    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d44f64c6-pwnwg_d6e63ebe-da2c-4833-aa6f-522980588a96/kube-rbac-proxy/0.log"
Sep 09 20:51:34.294023 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.293993    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/thanos-query/0.log"
Sep 09 20:51:34.331609 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.331582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/kube-rbac-proxy-web/0.log"
Sep 09 20:51:34.368606 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.368584    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/kube-rbac-proxy/0.log"
Sep 09 20:51:34.396115 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.396092    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/prom-label-proxy/0.log"
Sep 09 20:51:34.434739 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.434650    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/kube-rbac-proxy-rules/0.log"
Sep 09 20:51:34.469916 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:34.469893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-754fb6dd44-bmlxn_2885160f-0aa1-4e01-b64a-194e61e0c276/kube-rbac-proxy-metrics/0.log"
Sep 09 20:51:36.536860 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.536828    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cc859cdcc-xh6sf_211a4988-1056-48e7-8d0a-d277023a9110/console/0.log"
Sep 09 20:51:36.588149 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.588124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-g68c4_895a680f-2d1a-4c04-b9c9-7be40f8cd95c/download-server/0.log"
Sep 09 20:51:36.655552 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655521    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"]
Sep 09 20:51:36.655928 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655913    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kube-rbac-proxy"
Sep 09 20:51:36.655976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655930    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kube-rbac-proxy"
Sep 09 20:51:36.655976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655943    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container"
Sep 09 20:51:36.655976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655949    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container"
Sep 09 20:51:36.655976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655963    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="storage-initializer"
Sep 09 20:51:36.655976 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.655972    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="storage-initializer"
Sep 09 20:51:36.656140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.656035    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kube-rbac-proxy"
Sep 09 20:51:36.656140 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.656047    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be4340d-5b11-4fa2-808b-3f62629006c6" containerName="kserve-container"
Sep 09 20:51:36.659055 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.659038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.661603 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.661583    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5xxks\"/\"default-dockercfg-rdwtx\""
Sep 09 20:51:36.661747 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.661628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xxks\"/\"openshift-service-ca.crt\""
Sep 09 20:51:36.662467 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.662449    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xxks\"/\"kube-root-ca.crt\""
Sep 09 20:51:36.667505 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.667484    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"]
Sep 09 20:51:36.734257 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.734224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-lib-modules\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.734257 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.734268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-proc\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.734514 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.734303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpj6\" (UniqueName: \"kubernetes.io/projected/a9d7297a-fb48-4036-9279-70313dcc1f7a-kube-api-access-cvpj6\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.734514 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.734374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-podres\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.734514 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.734435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-sys\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-sys\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835202 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-lib-modules\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-sys\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-lib-modules\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-proc\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpj6\" (UniqueName: \"kubernetes.io/projected/a9d7297a-fb48-4036-9279-70313dcc1f7a-kube-api-access-cvpj6\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835403 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-proc\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-podres\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.835636 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.835531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9d7297a-fb48-4036-9279-70313dcc1f7a-podres\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.843598 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.843572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpj6\" (UniqueName: \"kubernetes.io/projected/a9d7297a-fb48-4036-9279-70313dcc1f7a-kube-api-access-cvpj6\") pod \"perf-node-gather-daemonset-bxtgh\" (UID: \"a9d7297a-fb48-4036-9279-70313dcc1f7a\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:36.970408 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:36.970371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:37.240515 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:37.240480    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"]
Sep 09 20:51:37.244014 ip-10-0-135-16 kubenswrapper[2582]: W0909 20:51:37.243987    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda9d7297a_fb48_4036_9279_70313dcc1f7a.slice/crio-dcc525a5a5af2d0d2bea53df0e822ae0f47f2e266ca70f27e37a8a12f9a3885d WatchSource:0}: Error finding container dcc525a5a5af2d0d2bea53df0e822ae0f47f2e266ca70f27e37a8a12f9a3885d: Status 404 returned error can't find the container with id dcc525a5a5af2d0d2bea53df0e822ae0f47f2e266ca70f27e37a8a12f9a3885d
Sep 09 20:51:38.139180 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.139135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh" event={"ID":"a9d7297a-fb48-4036-9279-70313dcc1f7a","Type":"ContainerStarted","Data":"5489ba35f8e8f684587cd00cddf83c23dd49ff0602909d39433b3930b721c3e2"}
Sep 09 20:51:38.139589 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.139194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh" event={"ID":"a9d7297a-fb48-4036-9279-70313dcc1f7a","Type":"ContainerStarted","Data":"dcc525a5a5af2d0d2bea53df0e822ae0f47f2e266ca70f27e37a8a12f9a3885d"}
Sep 09 20:51:38.139589 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.139250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:38.158363 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.158318    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh" podStartSLOduration=2.158305745 podStartE2EDuration="2.158305745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:38.156209333 +0000 UTC m=+3387.556886525" watchObservedRunningTime="2026-09-09 20:51:38.158305745 +0000 UTC m=+3387.558982984"
Sep 09 20:51:38.222256 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.222231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw5cx_d4cb9d2f-d6e8-402e-9e48-60589350d940/dns/0.log"
Sep 09 20:51:38.248631 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.248602    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw5cx_d4cb9d2f-d6e8-402e-9e48-60589350d940/kube-rbac-proxy/0.log"
Sep 09 20:51:38.272994 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.272970    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cn5pp_f145ad01-42a3-4ad7-a6cd-f556d7c99fca/dns-node-resolver/0.log"
Sep 09 20:51:38.816219 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.816178    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57b87587df-pzhtd_81eed42f-a908-4013-bbd4-ceaa766a77f4/registry/0.log"
Sep 09 20:51:38.913059 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:38.913027    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tsnld_0de05ec0-24fb-4500-a408-3cf08a5b210b/node-ca/0.log"
Sep 09 20:51:40.130560 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:40.130522    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq2pt_9311d573-337d-4a5f-9693-f675a92c0269/serve-healthcheck-canary/0.log"
Sep 09 20:51:40.612292 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:40.612261    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkddx_b99cff65-7625-4001-835b-52fe979042d4/kube-rbac-proxy/0.log"
Sep 09 20:51:40.634450 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:40.634427    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkddx_b99cff65-7625-4001-835b-52fe979042d4/exporter/0.log"
Sep 09 20:51:40.659779 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:40.659741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkddx_b99cff65-7625-4001-835b-52fe979042d4/extractor/0.log"
Sep 09 20:51:43.342987 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:43.342953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7676c798d9-dqs57_d98830e7-e434-41ec-81a1-0485483f2e82/manager/0.log"
Sep 09 20:51:43.395961 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:43.395935    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2xc2s_5325fcbe-4f01-4b6e-a7bd-7fedaf04a813/server/0.log"
Sep 09 20:51:43.891492 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:43.891453    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-dgbt7_f6a322b8-ca57-416b-9c43-3464c1b2b357/manager/0.log"
Sep 09 20:51:44.090101 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:44.090072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6kgl5_a8eac914-d728-49e6-913b-ec115d861f8b/seaweedfs/0.log"
Sep 09 20:51:44.116671 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:44.116641    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-xkg6j_ce703ffd-a1fc-4996-9694-7f648df0c9e5/seaweedfs-tls-custom/0.log"
Sep 09 20:51:44.152868 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:44.152810    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-bxtgh"
Sep 09 20:51:48.765420 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:48.765394    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-qnh44_5f05fec7-154a-4d14-8103-e2561e53a93f/migrator/0.log"
Sep 09 20:51:48.793031 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:48.792997    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-qnh44_5f05fec7-154a-4d14-8103-e2561e53a93f/graceful-termination/0.log"
Sep 09 20:51:49.148794 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:49.148712    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-r7ll8_9b080784-3270-422d-bced-001ff0632258/kube-storage-version-migrator-operator/1.log"
Sep 09 20:51:49.204243 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:49.204217    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-r7ll8_9b080784-3270-422d-bced-001ff0632258/kube-storage-version-migrator-operator/0.log"
Sep 09 20:51:50.331733 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.331702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:51:50.360174 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.360145    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/egress-router-binary-copy/0.log"
Sep 09 20:51:50.388627 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.388598    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/cni-plugins/0.log"
Sep 09 20:51:50.415775 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.415748    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/bond-cni-plugin/0.log"
Sep 09 20:51:50.440137 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.440106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/routeoverride-cni/0.log"
Sep 09 20:51:50.468531 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.468507    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/whereabouts-cni-bincopy/0.log"
Sep 09 20:51:50.501033 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.501012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cpkgx_5485e8a1-7d02-4313-84d1-d603d7016774/whereabouts-cni/0.log"
Sep 09 20:51:50.835747 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.835714    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cb56b_4d9be85a-318f-4ab5-aece-2c274cef457d/kube-multus/0.log"
Sep 09 20:51:50.916009 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.915979    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l4zfq_aab7313b-388e-42a1-8f32-3f2e4f594077/network-metrics-daemon/0.log"
Sep 09 20:51:50.937118 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:50.937090    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l4zfq_aab7313b-388e-42a1-8f32-3f2e4f594077/kube-rbac-proxy/0.log"
Sep 09 20:51:52.068022 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.067990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/ovn-controller/0.log"
Sep 09 20:51:52.087455 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.087429    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/ovn-acl-logging/0.log"
Sep 09 20:51:52.118228 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.118204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/kube-rbac-proxy-node/0.log"
Sep 09 20:51:52.142543 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.142518    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:51:52.170515 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.170492    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/northd/0.log"
Sep 09 20:51:52.198704 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.198658    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/nbdb/0.log"
Sep 09 20:51:52.223745 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.223718    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/sbdb/0.log"
Sep 09 20:51:52.292795 ip-10-0-135-16 kubenswrapper[2582]: I0909 20:51:52.292770    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr562_b805f433-8c13-41a2-8222-5b1caf76fdb2/ovnkube-controller/0.log"